Notes on Anthropic's Computer Use Ability • Hacker News Board

Re: Notes on Anthropic's Computer Use Ability

#41

Post by steveBK123 » Fri, Oct 25, 2024, 2:13 PM UTC

belval wrote: post #35

The product I would like to see out of this is a way to automate UI QA. Ideally it would be given a persona and a list of use cases, try to accomplish each task and save the state where you/it failed. Something like a Chrome lighthouse but for usability. Bonus point if it can highlight what part of my documentation is using mismatched terminology making it difficult for newcomers to understand what button I am referr…

I've seen similar sentiment even pre-LLM that AI would help automate other forms of testing, and I just don't quite see it.

Implementing tests is not the hard part. You could make that an intern project or hire a consultant for 3 months. The hard part is the interpretation of results.

That is - making a thing that spits out tickets/alerts is easy. The signal/noise tuning and actual investigation workflows are the hard part and still very manual & human operated. I don't see LLM mouse/keyboard control changing that yet.

Re: Notes on Anthropic's Computer Use Ability

#42

Post by elif » Fri, Oct 25, 2024, 2:14 PM UTC

alt-glitch wrote: post #38

I wonder if I can hook up `scrcpy` with this and give it control over an Android. Can it drag the mouse? That'd be needed to navigate the phone at least.

For my home automation system my aspiration is to give the AI control of an android virtual machine

Re: Notes on Anthropic's Computer Use Ability

#43

Post by [deleted] » Fri, Oct 25, 2024, 2:14 PM UTC

nilstycho wrote: post #29

It seems like a cheaper intermediate capability would be to give Claude the ability to SSH to your computer or to a cloud container. That would unlock a lot of possibilities, without incurring the cost of the vision model or the difficulty of cursor manipulation. Does this already exist? If not, would the benefits be lower than I think, or would the costs be higher than I think?

[deleted]

Re: Notes on Anthropic's Computer Use Ability

#44

Post by elif » Fri, Oct 25, 2024, 2:16 PM UTC

_heimdall wrote: post #18

I'm all for the MVP approach and shipping quickly, though I'm really surprised they went with image recognition and tooling for injecting mouse/keyboard events for automating human tasks. I wonder why leveraging accessibility tools for this wouldn't have been a better option. Browsers and operating systems both have pretty comprehensive tooling for accessibility tools like screen readers, and the whole point of those…

Crazy that this needs to be said but 'Computer use' is far more expansive of a domain than Internet browsing...

Re: Notes on Anthropic's Computer Use Ability

#45

Post by bamboozled » Fri, Oct 25, 2024, 2:16 PM UTC

tsunamifury wrote: post #37

im3w1l wrote: post #26

Earlier quoted context omitted.

Have you heard of Centaur chess? A human and a machine would team up to find the best chess moves against another similar team. It's not a thing anymore. Computers have advanced so much that humans can't really contribute in any meaningful sense.

Once we realize we can make machines that can beat us in ways we can’t even understand, I wonder if will question if we have always been influenced this way by an exterior force

Sounds like an interesting idea, do you mean, like the concept of "fate" is the type of external force you describe ?

Re: Notes on Anthropic's Computer Use Ability

#46

Post by reportgunner » Fri, Oct 25, 2024, 2:18 PM UTC

nilstycho wrote: post #39

reportgunner wrote: post #31

Earlier quoted context omitted.

Not just benefits, costs but also the risks are to be considered here I think.

What are the risks? Isn't this a strict subset of the risks of full desktop access? Claude can just open a GUI terminal with Computer Use. (I think.)

Software posing as Claude that is actually a malware tricking an unsuspecting non-terminal user into executing it is what I was thinking about.

Re: Notes on Anthropic's Computer Use Ability

#47

Post by bamboozled » Fri, Oct 25, 2024, 2:20 PM UTC

exe34 wrote: post #34

bamboozled wrote: post #30

Earlier quoted context omitted.

Is the point of your comment to make people feel depressed ? Either we're going to use these tools to augment our abilities or basically just become wiped out, at least our jobs will be, and there is no plan to provide support for anyone. Maybe the tech will make the transition to a post employment world so swift we don't even feel any negative economic effects at all, but let's see.

is this a cry for help? there's always alcohol and drugs, they can't take that away from us! (unless the robots of the future are like Bender)

I'm not saying I am depressed, but I mean, the comment just sounded like such a major downer.

Re: Notes on Anthropic's Computer Use Ability

#48

Post by apwell23 » Fri, Oct 25, 2024, 2:21 PM UTC

jampekka wrote: post #27

nerdix wrote: post #25

Earlier quoted context omitted.

It's main use case is making the average office worker feel the same existential dread that some programmers feel when they see a LLM spit out a bunch of code in mere seconds.

And when having to pick up the pieces when someone actually uses the thing.

Thats the real dread

Re: Notes on Anthropic's Computer Use Ability

#49

Post by belval » Fri, Oct 25, 2024, 2:22 PM UTC

steveBK123 wrote: post #41

belval wrote: post #35

The product I would like to see out of this is a way to automate UI QA. Ideally it would be given a persona and a list of use cases, try to accomplish each task and save the state where you/it failed. Something like a Chrome lighthouse but for usability. Bonus point if it can highlight what part of my documentation is using mismatched terminology making it difficult for newcomers to understand what button I am referr…

I've seen similar sentiment even pre-LLM that AI would help automate other forms of testing, and I just don't quite see it. Implementing tests is not the hard part. You could make that an intern project or hire a consultant for 3 months. The hard part is the interpretation of results. That is - making a thing that spits out tickets/alerts is easy. The signal/noise tuning and actual investigation workflows are the har…

> making a thing that spits out tickets/alerts is easy.

I don't really believe that what I am asking for is hard, yet I still can't buy it as far as I know.

> actual investigation workflows are the hard part and still very manual & human operated.

Sure but it would allow your QA worker to have pre-tested usecase-based path with some flag on whether or not they may be problematic with a screen-recording and some timestamp of where it went wrong.

These will always need human-in-the-loop to vet the findings before cutting a ticket to development team.

Re: Notes on Anthropic's Computer Use Ability

#50

Post by diffeomorphism » Fri, Oct 25, 2024, 2:23 PM UTC

soham123 wrote: post #13

utkarsh-dixit wrote: post #6

This is such a idiotic hype-cycle, they just fine-tuned a model over vision API. I really don't understand why everyone is loosing their mind over this

They showed a really cool literal example of what's coming. it's almost a chatgpt like movement.

Which one? The article has four examples, none of which are particularly "cool" or impressive.

If anything, the examples involving moving the mouse to the address bar or getting csv's of results are very poor examples, because we can already do that much better without "computer use".