you drove all the way to Albany then next morning head back to Margaret River?
Was Albany interesting at all?
Albany doesn't have a lot of things, but enough to keep you busy for a couple of hours walking around the shops and see the historical buildings amidst the modern structures.
The real charm is actually out of Albany. For natural wonders, you'll get blowhole, the gap (must go.. amazing place) and natural bridge.
There's also whaleworld, not very very fascinating, but you'll get to learn a lot about Albany's whaling history..
A short detour will bring you to windfarm. I went there just before sunset and I like it

I did the whale cruise in Albany, sadly just saw 1 whale only + some dolphins..
By the way, for lodging, I stayed in Vancouver Guest House in Albany. Location is fantastic, all the rooms deco are different, so you'll get to choose which room you like most. Trish (the owner) is a really nice warm lady, and her home-cooked breakfast made it the best lodging we've stayed in WA.