That's the usual criticism of Panerai. To be fair, there are several people in the finance industry that have fairly large egos (I was in the finance industry for a while. Front office, where the biggest egos are, typically. Haha!), and the brands favored by these folks are Omega, Rolex and Panerai.
Panerai is also criticized for the lack of heritage and in-house innovation. I know of the history, back in the days where they were used by the Italian military, but it really isn't much when compared to the other makes, especially since they have been buying movements from suppliers for much of their existence.
Nonetheless, I think the choice of a watch is a reflection of your personal tastes and the image you wish to portray. Some think a Panerai looks great, while others think it is a fad. I think as long as you like it, I wouldn't let all these "passionate" discussions discourage you.