The concept is good but accuracy in cluttered environment can be a concern, also misinterpreting context can be a problem
Reimagining the mouse pointer for the AI era
41–50 of 223 posts
Re: Reimagining the mouse pointer for the AI era
#42My reaction to the first demo (recipe) is that it was slower than typing the same thing on your keyboard.
The second demo seems to be a wash: there's no time saved in saying "move this" versus "move crab". And an app-specific contextual menu would probably be faster.
The third demo doesn't seem to warrant the use of a pointer at all, since there is only one way to interpret the prompt.
None of this means that this approach will not be successful, but there's a reason why so many attempts to revolutionize user interfaces ended up going nowhere. Talking to your computer was always supposed to be the future, but in practice, it's slower and more finicky than typing.
In fact, the only new UI paradigm of the past 28+ years appears to have been touchscreens and swipe gestures on phones. But they are a matter of necessity. No one wants to finger-paint on a desktop screen.
Re: Reimagining the mouse pointer for the AI era
#43Next generation of OS should have constant video and audio recognition by on device LLM. This will provide valuable context for a lot of scenarios. So instead of frequent copy-pasting we are used to, we can let agents access context of our whole workflows from different apps.
But Google is a very ill positioned candidate for such OS. I would rather trust Apple and local-first on-device models.
Re: Reimagining the mouse pointer for the AI era
#44My reaction to the first demo (recipe) is that it was slower than typing the same thing on your keyboard. The second demo seems to be a wash: there's no time saved in saying "move this" versus "move crab". And an app-specific contextual menu would probably be faster. The third demo doesn't seem to warrant the use of a pointer at all, since there is only one way to interpret the prompt. None of this means that this ap…
Talking to your computer can only ever work for people in atomized work-from-home silos, surely. I can't really imagine living in a world where everybody is just muttering commands to the computer all the time.
Re: Reimagining the mouse pointer for the AI era
#45This is how I always imagined FE development would work once ChatGPT 3 came out. Then Cursor appeared and seeing how successful they were with just a chat and a few tool calls, I thought I was over-complicating things.
Anyway, I built a prototype on this idea, but instead of relying only on hover, I press Option to select a node in a custom AST-ish semantic layer I designed around a minimalist UI grammar, and Option + up/down arrows to move to parent/child node. This way, I have have an accurate pointer to the element I want to talk about, plus a minimal context window (parent component, state, a few navigation related queries).
What I learned from using it, though, is that the killer use case isn't necessarily the flashy "talk to this UI element" interaction shown in the Google demos. I do use it that way too; I have `Option + Shift + click` to copy a selector to the clipboard, so I can give an LLM connected to the live medium a precise reference to the element I want to discuss.
But the place where it has been most useful day to day is much simpler: source navigation. Point at the thing in the UI, jump to the code that is responsible for it. The difficult part is jumping to the code you care about (the code for UI or for the semantic element?), but in my system that distinction turned out to be usually obvious, which is what makes the interaction useful.
Re: Reimagining the mouse pointer for the AI era
#46Both of the text based demos would have been simpler and faster with traditional mouse and keyboard interactions. What is the AI adding?
Yeah it's a gimmick
Re: Reimagining the mouse pointer for the AI era
#47what the hell is going on at google
AI.
Re: Reimagining the mouse pointer for the AI era
#48I sense a privacy problem brewing. It reminds me of Microsoft Recall in the sense that some portion of the screen is going to be continuously transmitted outside of the users control. What happens when someone browses something very private (planning a surprise engagement. looking at medical data. planning a protest)? All that data gets slurped to google and subject to a warrant or discovery or building your advertis…
> What happens when someone browses something very private?
Profit!
Re: Reimagining the mouse pointer for the AI era
#49Next generation of OS should have constant video and audio recognition by on device LLM. This will provide valuable context for a lot of scenarios. So instead of frequent copy-pasting we are used to, we can let agents access context of our whole workflows from different apps. But Google is a very ill positioned candidate for such OS. I would rather trust Apple and local-first on-device models.
Next generation OS should absolutely -not- have always-on surveillance like you describe.
Re: Reimagining the mouse pointer for the AI era
#50I've been doing something similar to this in a personal claude code frontend, though not particularly "magical".
I'm mostly using my system to make comments on long AI-generated documents (especially design documents). I find it works well to have the AI generate something, and then I read through it, making comments along the way.
You can get pretty far just repeating the things you see... "I'm reading [heading] and [comments]". But I do find some use in selecting content and saying "I don't agree with this" or whatever else.
The result is just an augmented message. It looks like:
Let's see what we've got here.
The system already...
No, I don't like how this is approaching the problem, ...
It makes reviewing feel pretty relaxed and natural. I've played around with similar note taking systems, which I think could be great for studying in school, but haven't had the focus on that particular problem to take it very far.
But I think the best thing really is giving the agent a richer understanding of what the user is experiencing and doing and just creating a rich representation of that. The keywords can be useful, but almost only as checkpoints: a keyword can identify the moment to take the transcript and package it up and deliver it.
One difference perhaps in design motivation: I have really embraced long latency interactions. I use ChatGPT with extended thinking by default, and just suck it up when the answer didn't really require thinking. I deliver 10 points of feedback at once instead of little by little. (Often halfway through I explicitly contradict myself, because I'm thinking out loud and my ideas are developing.) I just don't stress out about latency or feedback, and so low-latency but lower-intelligence interactions don't do it for me (such as ChatGPT's advanced voice mode, or probably Thinking Machine's work). I think this focus is in part a value statement: I'm trying to do higher quality work, not faster work.