Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reimagining the mouse pointer for the AI era

deepmind.google

Post reply on HN
Print view View original

171–180 of 223 posts

Re: Reimagining the mouse pointer for the AI era

#171

Post by hansmayer »

why_at wrote: post #28

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…

Well you see to really, really sell it to the common folks, they need to convince you that ChatBots are the "Intelligence" . So they are coming up with all sorts of crap, like this one. The TV advertisements for Gemini and co. are indicative of how they see the average user, as an idiot of sorts, who needs the shit-device for pretty much anything. Oh you spilled some water on the counter top? Quick, ask Gemini what to do! You are a 20ish something individual home alone? Quick, lay on the couch and ask Gemini if you can really talk to it, omg, its so exciting! You were in holidays all alone, but in the middle of a really large crowd? Gemini to the help, cut those people out and make it look like it was an exclusive spot, just for you! Nobody else was there. So this proposal is going into the same direction - probably targeting the average office "idiot".

Re: Reimagining the mouse pointer for the AI era

#172

Post by jdougan »

dgently7 wrote: post #138
AirMax98 wrote: post #37

Earlier quoted context omitted.

Right — it does seem cool but the voice is patching over a major gap. If I'm talking already, why wouldn't I just describe what I'm looking at and have the AI grab it for me?

pull up any moderately busy picture with more than a trivial amount of objects. pictures of "traffic" or with other similar repetition are great for this demo. pick one specific object (like a specific tire on one car) in the image and write (or say) out all the words youd need to specify that exact object. now take the same image and point at the object with your mouse or circle it with an annotation tool. its often…

Nothing new under the sun. "Put that there" demo, 1982.

https://www.media.mit.edu/publications/put-that-there-voice-...

https://www.youtube.com/watch?v=RyBEUyEtxQo

Re: Reimagining the mouse pointer for the AI era

#173

Post by PAndreew »

schnitzelstoat wrote: post #167
why_at wrote: post #28

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…

Yeah, I'd hate to use this in an open-plan office (which is like 99% of offices these days) and even using it alone at home would feel awkward. I don't really want to talk to the computer despite what 1950's sci-fi books led us to believe. It's a cool idea for the future when we have reliable EEG headsets or Neuralink or whatever though.

The only place I'd ever talk to a machine is my car. Instead of huge flashy screens that distracts and kills thousands of people maybe they could build a buttons + voice agent system that could actually be useful and durable. I hate to tap Waze/Maps/etc. every time when I go somewhere or that I cannot comfortably switch to specific songs en route without risking my life...

Re: Reimagining the mouse pointer for the AI era

#174

Post by RamblingCTO »

What a load of horsecrap. Google was never good at usability or UX. But that's a new low. This is ambiguous as it gets and good UX is opposite of that. If I need to undo half the stuff that happened or an AI starts to do stuff I don't want ot because I am moving my mouse in a certain way I'd just get angry and turn it off.

Re: Reimagining the mouse pointer for the AI era

#175

Post by dandaka »

mrhottakes wrote: post #49
dandaka wrote: post #43

Next generation of OS should have constant video and audio recognition by on device LLM. This will provide valuable context for a lot of scenarios. So instead of frequent copy-pasting we are used to, we can let agents access context of our whole workflows from different apps. But Google is a very ill positioned candidate for such OS. I would rather trust Apple and local-first on-device models.

Next generation OS should absolutely -not- have always-on surveillance like you describe.

why not? if context stays on device and operator is in full control, what downsides are there? there is no observer here, only operator and full context of his activity at his fingers

Re: Reimagining the mouse pointer for the AI era

#176

Post by dandaka »

DonHopkins wrote: post #109
Legend2440 wrote: post #102

Earlier quoted context omitted.

It's not surveillance if it's on-device.

https://en.wikipedia.org/wiki/Sousveillance >Sousveillance (/suːˈveɪləns/ soo-VAY-lənss) is the recording of an activity by a member of the public, rather than a person or organisation in authority, typically by way of small wearable or portable personal technologies.[14] The term, coined by Steve Mann,[15] stems from the contrasting French words sur, meaning "above", and sous, meaning "below", i.e. "surveillance" de…

The goal is not to record activities of others, it is to have full context of operator activity. More like a personal knowledge base, that camera pointing to a cop.

Re: Reimagining the mouse pointer for the AI era

#178

Post by schnitzelstoat »

PAndreew wrote: post #173
schnitzelstoat wrote: post #167

Earlier quoted context omitted.

Yeah, I'd hate to use this in an open-plan office (which is like 99% of offices these days) and even using it alone at home would feel awkward. I don't really want to talk to the computer despite what 1950's sci-fi books led us to believe. It's a cool idea for the future when we have reliable EEG headsets or Neuralink or whatever though.

The only place I'd ever talk to a machine is my car. Instead of huge flashy screens that distracts and kills thousands of people maybe they could build a buttons + voice agent system that could actually be useful and durable. I hate to tap Waze/Maps/etc. every time when I go somewhere or that I cannot comfortably switch to specific songs en route without risking my life...

I connect my iPhone to my car and it requires Siri to be enabled which I can then use to change songs, Google Maps destinations etc. without having to touch anything.

The Siri voice transcription is pretty awful compared to what I've experienced with ChatGPT though and it's weird going back almost to the pre-LLM world where you have to give such clear sort of computer-coded voice commands.

Re: Reimagining the mouse pointer for the AI era

#180

Post by jwx48 »

atq2119 wrote: post #145
anon84873628 wrote: post #101

Earlier quoted context omitted.

It seems that if we ultimately want to "move at the speed of thought," it will require speech.

There's the adage that writing is thinking, but even more accurately at least for me, editing is thinking. Neither typing speed nor dictation speed is a true bottleneck, but editing speech seems like it'd be harder than editing text. Though there may be some hybrid approach that can work well.

> editing is thinking.

I hadn’t realized until just now how accurate that is for me as well. Thank you.

Post reply on HN