Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reimagining the mouse pointer for the AI era

deepmind.google

Post reply on HN
Print view View original

101–110 of 223 posts

Re: Reimagining the mouse pointer for the AI era

#101

Post by anon84873628 »

concinds wrote: post #64
why_at wrote: post #28

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…

The second half of your comment is a go-to-market concern but doesn't feel so relevant for a research prototype. It could be done with a private local model too, maybe not by Google. But I don't think the voice problem is surmountable. I closed their image editing demo when I saw it required a mic. It would be appealing as a Spotlight-like text pop-up interface where you type instructions, which would work in social/…

It seems that if we ultimately want to "move at the speed of thought," it will require speech.

Re: Reimagining the mouse pointer for the AI era

#102

Post by Legend2440 »

mrhottakes wrote: post #49
dandaka wrote: post #43

Next generation of OS should have constant video and audio recognition by on device LLM. This will provide valuable context for a lot of scenarios. So instead of frequent copy-pasting we are used to, we can let agents access context of our whole workflows from different apps. But Google is a very ill positioned candidate for such OS. I would rather trust Apple and local-first on-device models.

Next generation OS should absolutely -not- have always-on surveillance like you describe.

It's not surveillance if it's on-device.

Re: Reimagining the mouse pointer for the AI era

#103

Post by fny »

why_at wrote: post #28

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…

It's possible to rely on mouth movements instead of sound. I've been tweaking visual speech recognition models (VSR) for the past few weeks so that I can "talk" to my agents at the office without pissing everyone off. It works okay. Limiting language to "move this" "clear that" along side context cues vastly simplifies the problem and makes it far more possible on device.

I think its brilliant UX.

Re: Reimagining the mouse pointer for the AI era

#104

Post by benwen »

jpatten wrote: post #20

Reminds me of Put That There https://m.youtube.com/watch?v=RyBEUyEtxQo

Beat me to it! Back in 1980. Here’s the official link https://www.media.mit.edu/publications/put-that-there-voice-...

And the paper https://dam-prod.media.mit.edu/uuid/8e6d934b-6c6f-48e4-b0a1-...

Re: Reimagining the mouse pointer for the AI era

#105

Post by DonHopkins »

jpatten wrote: post #20

Reminds me of Put That There https://m.youtube.com/watch?v=RyBEUyEtxQo

Ah yes, the spiritual progenitor of MacPlaymate.

https://en.wikipedia.org/wiki/MacPlaymate

>The game features a panic button that when clicked on will cover the computer screen with a fake spreadsheet. The player can also choose to print out Maxie's current pose as a pinup.

https://archive.org/details/mac_MacPlaymate

Geraldo interviews Chuck Farnham about getting sued by Playboy:

https://news.ycombinator.com/item?id=42571845

https://www.upi.com/Archives/1989/02/09/Playboy-sues-over-se...

Re: Reimagining the mouse pointer for the AI era

#106

Post by torben-friis »

Do you know the deep frustration of watching a tech illiterate person use a PC? When they type google in the omnibar and click on Google.com for every search, for example?

Now you get to hear every person in the office do that around you.

Like, good tech, but do googlers live in the real world? Do they genuinely like the idea of an open office full of people talking to their computers? Do they all live alone without human contact?

Re: Reimagining the mouse pointer for the AI era

#108

Post by Swizec »

anon84873628 wrote: post #101
concinds wrote: post #64

Earlier quoted context omitted.

The second half of your comment is a go-to-market concern but doesn't feel so relevant for a research prototype. It could be done with a private local model too, maybe not by Google. But I don't think the voice problem is surmountable. I closed their image editing demo when I saw it required a mic. It would be appealing as a Spotlight-like text pop-up interface where you type instructions, which would work in social/…

It seems that if we ultimately want to "move at the speed of thought," it will require speech.

> It seems that if we ultimately want to "move at the speed of thought," it will require speech.

Except for the large majority of people who read, type, and click way faster than they can talk. Especially for visual things it’s way faster to drag a rectangle than to describe what you want.

A lot of us also aren’t linear verbal thinkers. It would take minutes to hours to verbalize concepts we can grasp visually/schematically in seconds.

Great book on the topic: https://www.goodreads.com/book/show/60149558-visual-thinking

Re: Reimagining the mouse pointer for the AI era

#109

Post by DonHopkins »

Legend2440 wrote: post #102
mrhottakes wrote: post #49

Earlier quoted context omitted.

Next generation OS should absolutely -not- have always-on surveillance like you describe.

It's not surveillance if it's on-device.

https://en.wikipedia.org/wiki/Sousveillance

>Sousveillance (/suːˈveɪləns/ soo-VAY-lənss) is the recording of an activity by a member of the public, rather than a person or organisation in authority, typically by way of small wearable or portable personal technologies.[14] The term, coined by Steve Mann,[15] stems from the contrasting French words sur, meaning "above", and sous, meaning "below", i.e. "surveillance" denotes the "eye in the sky" watching from above, whereas "sousveillance" denotes bringing the means of observation down to human level, either physically (by mounting cameras on people rather than on buildings) or hierarchically (with ordinary people observing, rather than by higher authorities or by architectural means).[16][17][23]

Re: Reimagining the mouse pointer for the AI era

#110

Post by sipjca »

_def wrote: post #92
lorecore wrote: post #88

Earlier quoted context omitted.

Agreed. A lot of Google products feel anti-social (like Google glass). They are definitely missing a human touch. Perhaps a byproduct of elitism and leet code grind filtering of employees mixed with founder personality.

Nothing else expected - one of the examples in this very article marks some text in a doc and prompts "make this more human"

thats incredible
Post reply on HN