You're probably right…
To clarify though…
I don't expect to "stop" Larry or Serge (or anyone else) from walking through Delores Park with their Glass on while I'm there (at the same time, I fully expect the doorman at Zeitgeist to tell em "Sorry, you can't come in here with those"). An individual recording stuff they're seeing anyway is pretty much "their right" - what they then choose to do with that recording though, is just as much their responsibility. For example, if you record someone walking out of Good Vibrations or Crazy Horse, and you choose to upload that to YouTube to make fun of them, and that person then loses their job over it (perhaps at the church-owned school they work at?) - I'd like to see you held to account for that action (and held financially responsible for breaching someone else's privacy).
This is actually codified into my local (NSW, Australian) Law in part. It's legal to record (audio and/or video) any interaction you have with a police officer. It's _not_ legal to "publish" that recording without the officers consent. Practically, it's been determined that you can't introduce a recording of your interaction with police as evidence in court (at least without the recorded police's agreement), but you _can_ tell the judge you've got a recording that proves a police officer is lying in his testimony, and the judge can request to see the recording. This has become very well known amongst profiled and targeted groups (for me, many motorcycle riders indiscriminately targeted as "outlaw motorcycle gang members" routinely record all police interactions and several high-profile lawyers recommend it).
Where I think this become important - is when many such recordings are collected together. Larry and his Glass, you and your Glass - they have small scale privacy implications, and barring those users doing something obviously stupid with them (which is no different from cellphone camera "etiquette" - if you go hanging outside abortion clinics, mental health clinics, drug rehabilitation centers, or schools with your camera/phone/Glass - and distribute the pictures you capture, you should expect consequences). I think though that Google (or anybody else) should consider the collection of many many user's Glass images/video/audio to be a potential-nightmare privacy problem _for them_. Before they start archiving all that, I think someone should be asking the same sort of questions you'd ask the junior developer who says "I'm just logging all the form data, so we can use it for debugging later" - You'd be asking "Hang on, _all_ the form data? The personally identifying information? The credit card number/expiry/ccv? The social security number? The medical history? And you're just writing that all out in cleartext to the public webserver logfles? Ummm, nope. I've just revoked your commit credentials and powered down the webserver - you might want to gather your things, don't let the door hit you in the ass on the way out…"
I don't think it "shouldn't be allowed", but I do think people implementing wholesale gathering of "public space personally identifiable data" should be held to high standards of responsibility for having that collection - "High" as in "if we screw up and this data get misused/exposed/stolen, that'll be to a very high level of likelyhood a CEO/CTO/company-destroying event" standards. You don't "just collect all the health information you can", you don't "just collect all the credit card information you can", _I'd_ like public-space surveillance to be the same. I'd like mall operators to say "Of course we don't archive video surveillance - we've got procedures and policies in place to delete all recordings over 1 week old unless we get a written police request for specific cameras/times/persons-of-interest, and we have a monthly task that checks and ensures that policy is adhered to."