candid street shots

Yes, candid street shots have been around forever; everyone loves them. But I saw one today and a girl in the photo realized she was being photographed just at the moment the shutter was pressed. Her expression was one wishing she could disappear into thin air. Do we acknowledge at all that we are invading someone's privacy when we sneak a shot of them? By not giving them the opportunity to decline, are we assigning a rank of unimportance to them? I've never been very comfortable taking photos of people without their knowledge and permission and I am even less so now.