IIRC, fast lenses are typically made for the super professional, super rich, or super devoted guys and gals, who spare no expense to get the sharpest possible lens. I was eyeing, and eventually bought, the AF85/1.8, which was, to me, ideal as it is fast, reasonably priced, and boy, sharp, beautiful bokeh, etc. My pal Paul, on the other hand, can do no less than the AIS85/1.4, as that lens is, sad to admit, that slightly better than my AF85/1.8. Paul bought the 85/1.8.
Mostly the manufacturers put in more effort to create the super lens, that frequently will become a legend in their line-up, nevermind most people enticed into buying that brand will never buy these legends, but end up with pitiful kit lenses better serving as paper weight. Spare no expense also applies to the lens maker, so mostly, not all the time, these super fast lenses are sharper, mechanically better, more beautifully crafted, have better bokeh, less vignetting, etc etc, as no expense is spared.
As many bros and sis have mentioned, if you stop down 1 to 2 stops (I hazard to guess better 2 stops), you will be hard pressed to find the difference. That's why I settled on the 85/1.8, although the size of my wallet and the number of kids I have to put (eventually) through Uni also figured. But occassionally I will "splurge" on the bigger one, like I have the AF50/1.4, after owning a series of 50mm lenses with f1.4, f1.8, f2 (even one with f1.7 for my odd minolta x-700). Honestly I love the AI50/2 the best, but now-a-days convenience is more important and I use AF50/1.4.
So, if you must have the ultimate legendary lens that will usually provide that bit more in performance, go for it. If you pretty much want a good lens, you save more than half for sacrifising that half to 1 stop.
I am not venturing into the ego part of the equation as I fear repercussion. But honestly when I turn up in non-pro circle with the 85/1.8 you can see the people going ooo-aaa over such a big beautiful front element... There you go, even the not-so-fast prime can have an effect. The trick is to get the one lens with the biggest most impressive looking front (and side, gold letters do wonders!).
I am still eyeing the 180/2.8 after all these years....