Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reimagining the mouse pointer for the AI era

deepmind.google

Post reply on HN
Print view View original

51–60 of 223 posts

Re: Reimagining the mouse pointer for the AI era

#51

Post by xg15 »

AirMax98 wrote: post #37
why_at wrote: post #28

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…

Right — it does seem cool but the voice is patching over a major gap. If I'm talking already, why wouldn't I just describe what I'm looking at and have the AI grab it for me?

I think they answer that question pretty convincingly: Because if what you're looking at is already on the screen, it much more easy to point to it and say "that" than to describe it.

(And if it's an abstract entity like a file, it might not even be possible to describe it, short of rattling off the entire file path)

Re: Reimagining the mouse pointer for the AI era

#54

Post by layer8 »

mvdtnz wrote: post #8

Both of the text based demos would have been simpler and faster with traditional mouse and keyboard interactions. What is the AI adding?

The AI is adding that you don’t need to provide a button or menu item or keyboard shortcut for each possible action, or for the user having to know that the command is there and how to access it.

I prefer keyboard operation myself, but I can see how this could become useful in the future, for certain use cases.

What would be useful as well then is if you could bind such a repeatedly-used AI command to button/menu item/keyboard shortcut in a way that it can still be used with pointing “this” and “that”.

Re: Reimagining the mouse pointer for the AI era

#55

Post by botanrice »

while these examples might be easy fodder for criticism I do feel like this whole idea of talking to an LLM across multiple applications and anything your pointer is on will give it context is pretty powerful and cool idea.

I'm imagining a webpage with a link - instead of opening a new link to quickly google something or opening three new tabs based on hyperlinks, i can point at a paragraph or line and ask it to tell me about it.

Maybe I can point at a song on Spotify and have it find me the youtube video, or vice versa (of course this is assuming a tool like this wouldn't stay locked into one ecosystem.. which it will).

Point is that the concept of talking to the computer with mouse as pointer is pretty cool and i guess a step closer to that whole sci-fi "look at this part of the screen and do something"

Re: Reimagining the mouse pointer for the AI era

#56

Post by holoduke »

botanrice wrote: post #55

while these examples might be easy fodder for criticism I do feel like this whole idea of talking to an LLM across multiple applications and anything your pointer is on will give it context is pretty powerful and cool idea. I'm imagining a webpage with a link - instead of opening a new link to quickly google something or opening three new tabs based on hyperlinks, i can point at a paragraph or line and ask it to tell…

Yeah. We just need 10x more compute. But constant ai analysing of everything is the ultimate direction.

Re: Reimagining the mouse pointer for the AI era

#59

Post by robot-wrangler »

A zigzag merge gesture is obviously a terrible idea until/unless everything is a touch screen. Did they even think about this stuff at all? Ergonomics and RSI aside, if a horizontal drag means add, why not just make vertical drag mean merge. Not a fan of voice interaction generally, but it's something we'll all be grateful for as we get older. No need to accelerate it

Re: Reimagining the mouse pointer for the AI era

#60

Post by __MatrixMan__ »

I've been iterating on some 3D models for various wacky garage projects I have. It's fun. I've often wished I could click on an arbitrary place and say "add an eye bolt here" or somesuch.

Of course learning proper cad software is probably the right thing here, but having Claude write python scripts which generate HTML files which reference three.js to provide a 3d view has gotten me surprisingly far. If something could take my pointer click and reverse whatever coordinate transforms are between the source code and my screen such that the model sees my click in terms of the same coordinate system it's writing python in, well that would be pretty slick.

Post reply on HN