Reimagining the mouse pointer for the AI era • Hacker News Board

Re: Reimagining the mouse pointer for the AI era

#221

Post by botanrice » Wed, May 13, 2026, 5:52 PM UTC

holoduke wrote: post #56

botanrice wrote: post #55

while these examples might be easy fodder for criticism I do feel like this whole idea of talking to an LLM across multiple applications and anything your pointer is on will give it context is pretty powerful and cool idea. I'm imagining a webpage with a link - instead of opening a new link to quickly google something or opening three new tabs based on hyperlinks, i can point at a paragraph or line and ask it to tell…

Yeah. We just need 10x more compute. But constant ai analysing of everything is the ultimate direction.

I wonder what the smallest, still practical, application of this would be now if you could implement it locally on your machine assuming you have decent hardware.

Maybe something with the file system? Like hover over any file and instead of seeing a snapshot or some details/metadata you could get a quick 1-2 line sentence on what it is. I suppose you may want to have that saved somewhere as well to cache it... but I'm def not an expert.

This would solve the always difficult issue of finding that one document! I still have trouble with document search on both mac & windows OS sometimes.

Re: Reimagining the mouse pointer for the AI era

#222

Post by arjie » Wed, May 13, 2026, 7:34 PM UTC

skydhash wrote: post #195

arjie wrote: post #35

Oh interesting, this is very cool. At first I thought it was just focus-follows-mouse but it's more interesting. You have certain keywords trigger "add to prompt". Ignoring the voice functionality (which is admittedly crucial currently because other inputs currently take over focus), I've often wanted to just have a continuous conversation with the LLM as I 'point and click' (or tab over and select) at various things…

> I have some small amount of bias because I've always felt input-constrained on computers. I have to move my hands to go places and that's exasperating. I've tried head tracking, had a vim pedal for a while, and used tiling WMs, and things like this to aid but while my vim-fu is pretty good and I function inside things very well with it, my cross-application interface isn't Why not constrain your computing? It will…

I can only conclude that I don't have the programming + product chops to figure out what is truly the common element that can be automated.

Re: Reimagining the mouse pointer for the AI era

#223

Post by order-matters » Thu, May 14, 2026, 4:04 PM UTC

skydhash wrote: post #74

order-matters wrote: post #70

Earlier quoted context omitted.

yeah, i understand the frustration of needing to do all the communication through typing and clicking and that it can feel limiting - but i want the computer to be less demanding of my physical reality not more. i want to be able to talk to someone on the phone, work on something ith my hands, and still successfully manage my compute tasks. improvement can only be made by requiring less attention to hte screen and le…

> i want to be able to talk to someone on the phone, work on something with my hands, and still successfully manage my compute tasks. Maybe you can share a scenario for that one? I can’t figure a scenario where all of this needs to be true. It seems like a recipe for accidents.

well to start, im not sure what the magical new input would be that doesnt involve hands or voice, but for conversation lets assume its magic. the ideal is that i can use magic input to manage cloud compute through my phone, and that leaves me hands free to exercise and listen to music, or play video games, or hold a baby in a quiet room, and still manage tasks.

realistically what would be best is flexible inputs. voice input is often neutered by being only voice, or requiring a click activation and click to stop recording, is part of the trouble.

Getting a "magic" input is not as hard as it seems, if you reduce its input space as not needing to compete with keyboards and voice. a workflow could involve an assistant making suggestions and then the magic input needs only be a yes/no which could be head nod / shake, eye tracking & blinking, foot pedals, hand gestures

honestly though i think the real quality of life improvement is going to come from OS's enabling multiple focus windows to be active simultaneously with multiple input devices. like what i really want is a keyboard that can act as 10+ virtual keyboards with a way to change which one is in use, and then based on which one is in use the inputs go directly to an app that is in focus for that input. let my game controller stay inputting into the game while i type something, or toggle my voice input to talk to an AI and not transmit to discord while im doing that. or i just get two keywords and two mice and splitscreen games with m&k input with another person next to me to two instances of the game running (one on each of two monitors), or the mic recognizes my partners voice as a separate input from my voice and managed independently

there is so much that could be done with interface beyond layout and menus even regardless of AI, but AI related tools could help with mapping different voices into different virtual inputs, or recognizing keywords to do the same