Wah... i feel like joining you!!
I did something different...that was bush walk with an experienced guide. I can't drive so i rely on public bus transport -intercity. I stayed at this place call Te Anau. It is between Queenstown and Milford sound. From QT it takes about 2 hours to get to Te Anau. From Te Anau it takes another 2 hrs to get to Milford Sound.
Te Anau is a quiet town and its a good stop over and stay. Most of the accomodation there is quite new, cheap and its very clean. Majority stay at Queenstown and went direct to Milford Sound ( a long drive). Staying in Queenstown is more expensive too.
Here's some links for your information... Trips and Tramps was the one I joined for buswalk. You may also book through Real Journeys. The best time to go is Nov before school holiday, it should be cheaper.. but can be quite cold.
http://www.milfordtourswalks.co.nz/
http://www.teanaubackpackers.co.nz/fiordland_activities.html