Probably the worst 'scolding' i got was when shooting this guy in Perth, dressed in a fancy costume as he was walking down the street talking on his mobile phone. I did my usual trick to look less conspicious like shooting from the hip from a distance thinking he wasnt going to notice anyway cos he was busy talking. BUT he did notice me! He said 'No' and came over and asked me to delete the photos i took. I obliged and showed him. Then as we both walked off, I could hear him saying in a loud voice 'Ask next time, A**hole!'
Of course the experience bugged me for the rest of the day, especially cos i noticed he had a small DSLR hanging round his shoulder (fellow shooter!). I asked myself if i had to ask permission everytime i wanted to photograph someone?
My own conclusion is No.
Oftentimes its not possible nor desirable to ask either from lack of time or fear of getting a posed look. Accept the fact that sometimes some people will take offense and others not. Heck, I would too if somebody started photographing me and i wasnt in the mood!
The point is not about offending someone, or 'feeling good' or being 'comfortable'. Its about the shot, and capturing the moment. If you love what you're doing, and the shot is worth the risk, and depending on how you feel, then take the shot. Just treat these times as lessons and 'battlescars' and get on with it. Refine your technique, or smile more, or do the business card thing.
Btw, i forgot to say 'sorry' to the guy which probably caused him to be more aggressive. I still managed to get a shot of him after he walked some distance.
2cents.