It also seems rude to be wearing a headset while socializing. People take their headphones out of their ears. If you have a bluetooth headset, I feel like I'm not getting your full attention. Even if you're not on a call, I still feel like I come second. It would be the same with Glass. If I'm talking to someone and they're wearing Glass, it seems rude. If they glance to the screen, even more so.
Then, it's unnecessary at times like you said. I think this is what looks smug, you're walking around with something on your head, that you might not be actively using. Or with bluetooth, you have an earpiece, but you're not in a call. It looks like you're wearing it, just to show it off, and it seems like you're trying hard to impress. With headphones, we make the assumption you're listening to music when they're in your ear.
People enjoy their technology, but they like to pretend they're free from it. A friend posted on Facebook the other day how everyone is always on their tablet or laptop, and 'gah, can't people just put their phone or tablet down for one day and look up'. But... she was on Facebook writing that message. I get that feeling from a lot of people. They keep their phone close, but they don't want to look like they depend on it. They want instant e-mail notifications or texts, but, they don't want to give off that image. It's not an attractive quality to always be playing with your phone or attached at the hip with technology. It's anti-social. That's a negative. When you're wearing something like Glass, you give off that impression.
And lastly, with Glass, you're literally pointing a camera at my face, and could be taking photos, and recording audio or video without my knowledge. That's a little uncomfortable.
Personally, I think it's going to fail, and I wouldn't be surprised if I never see a single person wearing the device. However, I applaud them for taking the risk, since the technology and advances will be used in some way. Could it be used by doctors to stay hands free? Could it be used on motorcycles, so you can see your speed or a rearview image in the corner of your eye? Does it work well just for around the house, so I can do housework, and get notifications? Or can I cook with dirty hands, and see my recipe, or cooking time remaining? I think there are uses for it, I think others will see what they've accomplished and apply it in some way. However, as a device that everyone will be wearing while walking down the street, I just don't see it happening.