Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reimagining the mouse pointer for the AI era

deepmind.google

Post reply on HN
Print view View original

141–150 of 223 posts

Re: Reimagining the mouse pointer for the AI era

#141

Post by NDlurker »

aquariusDue wrote: post #73
concinds wrote: post #64

Earlier quoted context omitted.

The second half of your comment is a go-to-market concern but doesn't feel so relevant for a research prototype. It could be done with a private local model too, maybe not by Google. But I don't think the voice problem is surmountable. I closed their image editing demo when I saw it required a mic. It would be appealing as a Spotlight-like text pop-up interface where you type instructions, which would work in social/…

This will sound like another brick in the paved road to dystopia but I'm kinda bullish on equipment that can recognize subvocalization. Or at least let me have a small drawing tablet with a stylus (think etch-a-sketch or Wacom Intuos) because at this point I'd rather practice writing and do away with typing altogether (even though I enjoy typing for typing's sake via MonkeyType).

I've been dreaming about that for 20 years. And then use it for people to communicate while sleeping.

Re: Reimagining the mouse pointer for the AI era

#142

Post by Melatonic »

shostack wrote: post #139
websap wrote: post #130

Earlier quoted context omitted.

You should look into how often people are using tools like WisprFlow and SuperWhisper. Voice is a very native mechanism. Most people working in open floor plans are wearing headphones any way. As long as you're not screaming, it's probably fine. Maybe, we'll move away from open plan offices in the bid for efficiency, which I would welcome.

I am moving full remote because dictation is such a better input mechanism for most of my AI interactions that I have become less efficient sitting in my open floorplan desk at the office because I cannot dictate there and the latency adds up. Typing is just achingly slow these days.

I feel like I can type faster than I can talk but I could be totally wrong?

Re: Reimagining the mouse pointer for the AI era

#143

Post by encom »

makeitdouble wrote: post #116
fny wrote: post #103

Earlier quoted context omitted.

It's possible to rely on mouth movements instead of sound. I've been tweaking visual speech recognition models (VSR) for the past few weeks so that I can "talk" to my agents at the office without pissing everyone off. It works okay. Limiting language to "move this" "clear that" along side context cues vastly simplifies the problem and makes it far more possible on device. I think its brilliant UX.

No UX needs to be perfect for everyone, but this doesn't sound trivial to make reliable. First things that came to mind: - facial hair - getting people to learn to make bigger mouth movements and not mumble - we're constantly self-correcting our speech as we hear our voice. This removes the feedback loop. - non english languages (god forbid bilingualism) - camera angles and head movement And that thinking about it fo…

>non english languages (god forbid bilingualism)

In my experience, any combination of computers + speech + danish has, so far without exception been terrible. Last time I tested ChatGPT, it couldn't understand me at all. I spoke both in my local dialect and as close to Rigsdansk [π] as I could manage. Unusable performance, and in any case I should be able to talk normally, or there's no point. It was about a year ago - it may have improved but I doubt it. I'm completely done trying to talk to machines.

Pre-emptive kamelåså: https://www.youtube.com/watch?v=s-mOy8VUEBk

[π] https://en.wikipedia.org/wiki/Danish_language#Dialects

Re: Reimagining the mouse pointer for the AI era

#145

Post by atq2119 »

anon84873628 wrote: post #101
concinds wrote: post #64

Earlier quoted context omitted.

The second half of your comment is a go-to-market concern but doesn't feel so relevant for a research prototype. It could be done with a private local model too, maybe not by Google. But I don't think the voice problem is surmountable. I closed their image editing demo when I saw it required a mic. It would be appealing as a Spotlight-like text pop-up interface where you type instructions, which would work in social/…

It seems that if we ultimately want to "move at the speed of thought," it will require speech.

There's the adage that writing is thinking, but even more accurately at least for me, editing is thinking.

Neither typing speed nor dictation speed is a true bottleneck, but editing speech seems like it'd be harder than editing text.

Though there may be some hybrid approach that can work well.

Re: Reimagining the mouse pointer for the AI era

#146

Post by el_benhameen »

Melatonic wrote: post #142
shostack wrote: post #139

Earlier quoted context omitted.

I am moving full remote because dictation is such a better input mechanism for most of my AI interactions that I have become less efficient sitting in my open floorplan desk at the office because I cannot dictate there and the latency adds up. Typing is just achingly slow these days.

I feel like I can type faster than I can talk but I could be totally wrong?

I also feel this way, but more importantly, I feel like my sentences are more coherent when typed because typing allows for corrections and modifications of ideas. Do whispr people just … get coherent, finalized ideas out in a single shot without any misspoken words?

Re: Reimagining the mouse pointer for the AI era

#148

Post by fatata123 »

ImaCake wrote: post #93

I think this falls flat for a technical audience because we already know how to do this stuff. But there are a lot of people who don't know how to copy paste, or use reverse image search, or apply a filter to a table. Being able to use plain language to do these things is a game changer for them. Sure, it's inefficient and inelegant but it's an interface that will do for basic technical stuff what the ipad touch scre…

[dead]

Re: Reimagining the mouse pointer for the AI era

#149

Post by jasonjayr »

why_at wrote: post #28

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…

The killer app was conceived as early as the 1980s: an agent running on your computer, organizing your files, your schedule, your messages, your bills, bank accounts, etc. All the parts of your life that were routine drudgery should be able to be offloaded to a smart agent, based on your preference, to bring you the information you needed with natural language queries, contextualized to what you were doing at the time, when you need it.

What's being delivered now is, an agent running on someone else's computer, copying your data to someone else's database, with zero responsibility, or mandate to protect that data and not share with with anyone else (in fact, they almost always promise to share it with their thousand partners), offering suggestions and preferences based on someone else's so-called recommendations, influenced by paying the agent's operators, and increasing pressure to make using someone else's computers + agents the only way to interact with other people and systems.

There is no doubt that LLM's can do amazing things, but the current environment seems to make it nearly impossible to do anything with them that doesn't let someone else inspect, influence, and even restrict everything you are doing with with these systems.

Re: Reimagining the mouse pointer for the AI era

#150

Post by gtowey »

el_benhameen wrote: post #146
Melatonic wrote: post #142

Earlier quoted context omitted.

I feel like I can type faster than I can talk but I could be totally wrong?

I also feel this way, but more importantly, I feel like my sentences are more coherent when typed because typing allows for corrections and modifications of ideas. Do whispr people just … get coherent, finalized ideas out in a single shot without any misspoken words?

They are not.

It's like a hidden curse of LLMs -- they're so good at parsing intended meaning from non-grammatically-correct language that we don't have to be very good at clear communication.

Eventually all LLMs will be controlled by humans uttering terse gutteral grunts. We will all become neanderthals, with machines that deliver our every whim.

Post reply on HN