"... surprisingly difficult and scary to climb compared to the Angkor Wat and Bapuon Temples due to the lack of hand-rails and the original laterite steps which can be very high and narrow."
There's a reason for that - the temples were never built for human beings! They were always meant to be the houses for the gods, so many of the staircases were decorative, not functional. You'll find that most temples have only one 'proper' set of stairs, which is usually on the east face!