Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reimagining the mouse pointer for the AI era

deepmind.google

Post reply on HN
Print view View original

191–200 of 223 posts

Re: Reimagining the mouse pointer for the AI era

#191

Post by skydhash »

jeswin wrote: post #151
jasonjayr wrote: post #149

Earlier quoted context omitted.

The killer app was conceived as early as the 1980s: an agent running on your computer, organizing your files, your schedule, your messages, your bills, bank accounts, etc. All the parts of your life that were routine drudgery should be able to be offloaded to a smart agent, based on your preference, to bring you the information you needed with natural language queries, contextualized to what you were doing at the tim…

A few decades back, a lot of computer use was emails. And it was stored on someone else's servers - with everyone from server operators along the route, to the government potentially having access to it. Even HTTPS is a relatively recent thing. I guess what I'm saying is - we've always had this problem.

Snail mail is also not secure and can be tampered with. I don’t mind someone hosting my mail. But I do mind Google doing it (based on their behavior).

Re: Reimagining the mouse pointer for the AI era

#192

Post by raincole »

chromacity wrote: post #42

My reaction to the first demo (recipe) is that it was slower than typing the same thing on your keyboard. The second demo seems to be a wash: there's no time saved in saying "move this" versus "move crab". And an app-specific contextual menu would probably be faster. The third demo doesn't seem to warrant the use of a pointer at all, since there is only one way to interpret the prompt. None of this means that this ap…

> My reaction to the first demo (recipe) is that it was slower than typing the same thing on your keyboard.

For you and me, who have used keyboard in our lives for more than 1,000 or even 10,000 hours.

There was a brief period when typing slowed people down because they could write the same information down with pen&paper, and that period eventually passed.

Re: Reimagining the mouse pointer for the AI era

#193

Post by Animats »

jasonjayr wrote: post #149
why_at wrote: post #28

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…

The killer app was conceived as early as the 1980s: an agent running on your computer, organizing your files, your schedule, your messages, your bills, bank accounts, etc. All the parts of your life that were routine drudgery should be able to be offloaded to a smart agent, based on your preference, to bring you the information you needed with natural language queries, contextualized to what you were doing at the tim…

> What's being delivered now is, an agent running on someone else's computer, copying your data to someone else's database, with zero responsibility, or mandate to protect that data and not share with with anyone else (in fact, they almost always promise to share it with their thousand partners), offering suggestions and preferences based on someone else's so-called recommendations, influenced by paying the agent's operators, and increasing pressure to make using someone else's computers + agents the only way to interact with other people and systems.

If we're going to have AI regulation, this is where to start. If a company's AI service acts for a user, the company has non-disclaimable financial responsibility for anything that goes wrong. There's an area of law called "agency", which covers the liability of an employer for the actions of its employees. The law of agency should apply to AI agents. One court already did that. An airline AI gave wrong but reasonable sounding advice on fares, a customer made a decision based on that advice, and the court held that the AI's advice was binding on the company, even though it cost the company money.

This is something lawyers and politicians can understand, because there's settled law on this for human agents.

Re: Reimagining the mouse pointer for the AI era

#195

Post by skydhash »

arjie wrote: post #35

Oh interesting, this is very cool. At first I thought it was just focus-follows-mouse but it's more interesting. You have certain keywords trigger "add to prompt". Ignoring the voice functionality (which is admittedly crucial currently because other inputs currently take over focus), I've often wanted to just have a continuous conversation with the LLM as I 'point and click' (or tab over and select) at various things…

> I have some small amount of bias because I've always felt input-constrained on computers. I have to move my hands to go places and that's exasperating. I've tried head tracking, had a vim pedal for a while, and used tiling WMs, and things like this to aid but while my vim-fu is pretty good and I function inside things very well with it, my cross-application interface isn't

Why not constrain your computing? It will require some programming chops, but you can note down your common tasks, figure out where actual input are required, and automate the rest.

Re: Reimagining the mouse pointer for the AI era

#196

Post by skydhash »

schnitzelstoat wrote: post #169
lorecore wrote: post #88

Earlier quoted context omitted.

Agreed. A lot of Google products feel anti-social (like Google glass). They are definitely missing a human touch. Perhaps a byproduct of elitism and leet code grind filtering of employees mixed with founder personality.

I can actually see more use-cases for Google Glass tbh - having a HUD would be useful when using Google Maps for example, like a fighter pilot. And being able to take photos/videos with the glasses (like the Meta ones nowadays) is really useful with my kid because he often does funny or cute stuff and I don't have time to pull my phone out to take a video/photo of it. I guess it could be useful for video calls too so…

> he often does funny or cute stuff and I don't have time to pull my phone out to take a video/photo of it.

IMO, that’s the anti social part. Why is a phone butting in your relation with your kid? I only have a few pictures taken when I was a kid (and most were for some grand occasion). And I’m happy that was the case because of so many cringeworthy things I did.

Re: Reimagining the mouse pointer for the AI era

#197

Post by swiftcoder »

fny wrote: post #103
why_at wrote: post #28

My first impression coming away from this is skepticism. Anything with voice controls for routine use is a pretty tough sell. Doing this when you're not completely alone would be annoying to everyone around you. Most of their examples seem like they could have been done with a right click drop down menu so they don't really need to "re-invent the mouse pointer". So is this thing talking to Google's servers all the ti…

It's possible to rely on mouth movements instead of sound. I've been tweaking visual speech recognition models (VSR) for the past few weeks so that I can "talk" to my agents at the office without pissing everyone off. It works okay. Limiting language to "move this" "clear that" along side context cues vastly simplifies the problem and makes it far more possible on device. I think its brilliant UX.

> I've been tweaking visual speech recognition models (VSR) for the past few weeks so that I can "talk" to my agents at the office without pissing everyone off.

Wouldn't SilentWhisper do just as good a job?

Re: Reimagining the mouse pointer for the AI era

#199

Post by iandanforth »

Deepmind hype is the worst hype. They do genuinely cool stuff and talk about it publicly, but don't make it available. Or it's only available to a tiny select few. Just shut up about the things you're doing until they are ready. You're part of a consumer products company, not a university PR department.

Re: Reimagining the mouse pointer for the AI era

#200

Post by lorecore »

schnitzelstoat wrote: post #169
lorecore wrote: post #88

Earlier quoted context omitted.

Agreed. A lot of Google products feel anti-social (like Google glass). They are definitely missing a human touch. Perhaps a byproduct of elitism and leet code grind filtering of employees mixed with founder personality.

I can actually see more use-cases for Google Glass tbh - having a HUD would be useful when using Google Maps for example, like a fighter pilot. And being able to take photos/videos with the glasses (like the Meta ones nowadays) is really useful with my kid because he often does funny or cute stuff and I don't have time to pull my phone out to take a video/photo of it. I guess it could be useful for video calls too so…

It's anti-social because other people don't want to be filmed by you.
Post reply on HN