I wonder what sort of monstrous power would be unleashed if Google used Plan9 as a foundation.
They'd half-finish it then bury it, like they did with Fuchsia which is heavily Plan-9-inspired.
Google Plan 4.5
Reimagining the mouse pointer for the AI era
111–120 of 223 posts
Re: Reimagining the mouse pointer for the AI era
#112All these problems dont need a conversation with an AI to solve. If you make the text selectable then you can do these actions fast and efficiently. This only becomes a product as they make the web shittier and less friendly towards PC workflows.
Its wild that they even put this out as a demo. It should have been picked apart in the internal meeting. There is no way I'd ever show my product taking 5s to change a 1 to a 2 in a piece of text that the user was already hovering or taking 10s to drag and drop a line of text from one box to another. Even the image of finding a route between two images could be done quick if images were auto OCR'd which is a setting on most image viewers.
Re: Reimagining the mouse pointer for the AI era
#113My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…
A General-Purpose Bubble Cursor
https://www.youtube.com/watch?v=46EopD_2K_4
>We present a general-purpose implementation of Grossman and Balakrishnan's Bubble Cursor [broken link] the fastest general pointing facilitation technique in the literature. Our implementation functions with any application on the Windows 7 desktop. Our implementation functions across this infinite range of applications by analyzing pixels and by leveraging human corrections when it fails.
Transcript:
>We present the general purpose implementation of the bubble cursor. The bubble cursor is an area cursor that expands to ensure that the nearest target is always selected. Our implementation functions on the Windows 7 desktop and any application for that platform. The bubble cursor was invented in 2005 by Grossman and Balakrishnan. However a general purpose implementation of this cursor one that works with any application on a desktop has not been deployed or evaluated. In fact the bubble cursor is representative of a large body of target aware techniques that remain difficult to deploy in practice. This is because techniques like the bubble cursor require knowledge of the locations and sizes of targets in an interface. [...]
https://www.dgp.toronto.edu/~ravin/papers/chi2005_bubblecurs...
>The Bubble Cursor: Enhancing Target Acquisition by Dynamic Resizing of the Cursor’s Activation Area
>Tovi Grossman, Ravin Balakrishnan; Department of Computer Science; University of Toronto
I've written more about Morgan Dixon's work on Prefab (pre-LLM pattern recognition, which is much more relevent with LLMs now).
https://news.ycombinator.com/item?id=11520967
https://news.ycombinator.com/item?id=14182061
Re: Reimagining the mouse pointer for the AI era
#114Both of the text based demos would have been simpler and faster with traditional mouse and keyboard interactions. What is the AI adding?
latency, cost and undefined behavior
Re: Reimagining the mouse pointer for the AI era
#115Both of the text based demos would have been simpler and faster with traditional mouse and keyboard interactions. What is the AI adding?
Maybe there’s potential here for certain accessibility scenarios for users who would benefit. But otherwise, leave the mouse cursor alone.
Re: Reimagining the mouse pointer for the AI era
#116My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…
It's possible to rely on mouth movements instead of sound. I've been tweaking visual speech recognition models (VSR) for the past few weeks so that I can "talk" to my agents at the office without pissing everyone off. It works okay. Limiting language to "move this" "clear that" along side context cues vastly simplifies the problem and makes it far more possible on device. I think its brilliant UX.
No UX needs to be perfect for everyone, but this doesn't sound trivial to make reliable.
First things that came to mind:
- facial hair
- getting people to learn to make bigger mouth movements and not mumble
- we're constantly self-correcting our speech as we hear our voice. This removes the feedback loop.
- non english languages (god forbid bilingualism)
- camera angles and head movement
Re: Reimagining the mouse pointer for the AI era
#117Their video demo is interesting. If that was to be useful, it would need to work on sites like Netflix. And for that to work, they would presumably have to axe drm. I am fully in favour of removing the pointless energy tax we pay as a society for the highly flawed and ineffective system of video drm.
Unless of course, their AI gets the same special privileges as the gpu in accessing drm content, and everything else is still locked out.
Re: Reimagining the mouse pointer for the AI era
#118Earlier quoted context omitted.
It seems that if we ultimately want to "move at the speed of thought," it will require speech.
> It seems that if we ultimately want to "move at the speed of thought," it will require speech. Except for the large majority of people who read, type, and click way faster than they can talk. Especially for visual things it’s way faster to drag a rectangle than to describe what you want. A lot of us also aren’t linear verbal thinkers. It would take minutes to hours to verbalize concepts we can grasp visually/schema…
Most people speak at about 150 wpm, but very few can type that fast. But reading and gesturing are fast, which is what TFA is about, combining reading and gesturing with speech.
Re: Reimagining the mouse pointer for the AI era
#119I like the idea of a touch screen that you don't touch. Just a few centimeters(variable) off would be fine - they do exist but I've never used one. I think what's required is a way to slightly flex one finger to activate cursor movement, then pinky/thumb twitch to button press. Maybe wear two magnetic silicone nail bands? Current air-touch technologies seem complex and power-hungry, I don't know.
Would be tiresome though to hold hand out all day - but good for mobile and handwriting/drawing. Need zero latency.
Furthermore, the mousepad could become the magnetic sensor and not the screen in order to rest the palm. The nail bands then become the equivalent of the mouse so it's a hall effect mouse. But could the pad detect finger twitch for the buttons, though?
Re: Reimagining the mouse pointer for the AI era
#120you can really tell the people building these tools spend a lot of time alone. I work from a home office 90% of the time and I wouldn't want this to be my workflow. I don't want to talk to my computer, I want to listen to music while I work, and I want to not sound deranged and disturb everyone around me when I am working from a coffeeshop or the open-plan office or the airport or the train or whatever. and that's as…
yeah, i understand the frustration of needing to do all the communication through typing and clicking and that it can feel limiting - but i want the computer to be less demanding of my physical reality not more. i want to be able to talk to someone on the phone, work on something ith my hands, and still successfully manage my compute tasks. improvement can only be made by requiring less attention to hte screen and le…
Yeah, I talk to someone on google meet who will seamlessly transition between talking to me and talking to Claude while on the call, and it is extremely annoying.