Live data from Hacker News

Reimagining the mouse pointer for the AI era

deepmind.google

121–130 of 223 posts

Re: Reimagining the mouse pointer for the AI era

#122
post #44

Earlier quoted context omitted.

Talking to your computer can only ever work for people in atomized work-from-home silos, surely. I can't really imagine living in a world where everybody is just muttering commands to the computer all the time.

This happens daily in radiology departments around the world

Aren’t radiologists dictating notes rather than issuing commands to the computer? From supporting them for a few years, I recall them having pretty good facility using the computer to zoom/filter/isolate parts of images, and most of the muttering was speech to text or a good old tape recorder for their writeups.

Re: Reimagining the mouse pointer for the AI era

#123
post #108

Earlier quoted context omitted.

> It seems that if we ultimately want to "move at the speed of thought," it will require speech. Except for the large majority of people who read, type, and click way faster than they can talk. Especially for visual things it’s way faster to drag a rectangle than to describe what you want. A lot of us also aren’t linear verbal thinkers. It would take minutes to hours to verbalize concepts we can grasp visually/schema…

Most people speak at about 150 wpm, but very few can type that fast. But reading and gesturing are fast, which is what TFA is about, combining reading and gesturing with speech.

You rarely need 150wpm when typing. If you try dictation, you’ll notice that half those words are error correction and checksum bits and just turn taking filler.

I usually convey the same meaning with 80wpm typing. Makes it faster to read too

Maybe I’m just slightly adhd – listening to people talk drives my crazy. Get to the point! Much easier if they type it out

Re: Reimagining the mouse pointer for the AI era

#124
Kind of incredible how consistently terrible Google is at everything they do in the AI space. So they choose this demo and write a big blog post and advertise.. and the demo is horrible, doesn't work. Doesn't track what I circle with the mouse, just apparently where the mouse pointer landed at the end and only exactly where it landed. Multiple times it said "Got it, I'll move this empty space between the clouds over here" or "Got it, I'll convert this empty area to a sunhat" despite my mouse only being a few pixels next to an actual hat.

Re: Reimagining the mouse pointer for the AI era

#125
post #63

you can really tell the people building these tools spend a lot of time alone. I work from a home office 90% of the time and I wouldn't want this to be my workflow. I don't want to talk to my computer, I want to listen to music while I work, and I want to not sound deranged and disturb everyone around me when I am working from a coffeeshop or the open-plan office or the airport or the train or whatever. and that's as…

Agreed. A lot of Google products feel anti-social (like Google glass). They are definitely missing a human touch. Perhaps a byproduct of elitism and leet code grind filtering of employees mixed with founder personality.

somebody the other day described google products as being a 22 year old imagining what their dad would want, and that feels very true to me.

Re: Reimagining the mouse pointer for the AI era

#126

while these examples might be easy fodder for criticism I do feel like this whole idea of talking to an LLM across multiple applications and anything your pointer is on will give it context is pretty powerful and cool idea. I'm imagining a webpage with a link - instead of opening a new link to quickly google something or opening three new tabs based on hyperlinks, i can point at a paragraph or line and ask it to tell…

I agree that AI audio interfaces will be the future but not because they are better UIs for users as we understand the term "user" today. The future users of UI are not users of UI at all, they want nothing to do with learning UI or what buttons to press or where to type something. They want to go to the shopping site and instead of typing anything into a search field they want to say "Find me some boots for the summer, I wanna look fresh" and then tell it to complete the purchase via voice as well once it found something. At most they'll still click some filters in the results page and on individual results but that will be it.

Re: Reimagining the mouse pointer for the AI era

#127

Earlier quoted context omitted.

yeah, i understand the frustration of needing to do all the communication through typing and clicking and that it can feel limiting - but i want the computer to be less demanding of my physical reality not more. i want to be able to talk to someone on the phone, work on something ith my hands, and still successfully manage my compute tasks. improvement can only be made by requiring less attention to hte screen and le…

> i want to be able to talk to someone on the phone, work on something with my hands, and still successfully manage my compute tasks. Maybe you can share a scenario for that one? I can’t figure a scenario where all of this needs to be true. It seems like a recipe for accidents.

It's not exactly "compute tasks" in this sense, but online gaming is similar. People are constantly talking to each other while managing multiple other kb/mouse inputs at the same time.

Re: Reimagining the mouse pointer for the AI era

#128
post #28

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…

Yup - what google is suggesting here will never materialize beyond being a slopfeature. People who want these bespoke workflows will build them or seek out specific tools that enable them, not trusting some overarching daemon that contextually watches their cursor. I don't trust google one bit to execute correctly on something like this.

Re: Reimagining the mouse pointer for the AI era

#129
post #8

Both of the text based demos would have been simpler and faster with traditional mouse and keyboard interactions. What is the AI adding?

It seems like the natural extension of the (unfortunate) simplification of UIs where everything is "cleanly" hidden behind multiple menus if not removed.

Re: Reimagining the mouse pointer for the AI era

#130
post #28

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…

You should look into how often people are using tools like WisprFlow and SuperWhisper. Voice is a very native mechanism. Most people working in open floor plans are wearing headphones any way. As long as you're not screaming, it's probably fine. Maybe, we'll move away from open plan offices in the bid for efficiency, which I would welcome.
Post reply on HN