Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on Anthropic's Computer Use Ability

composio.dev

Post reply on HN
Print view View original

81–90 of 108 posts

Re: Notes on Anthropic's Computer Use Ability

#81

Post by imranq »

supriyo-biswas wrote: post #77
imranq wrote: post #73

This is basically RPA with LLMs. And RPA is basically the worst possible solution to any problem. Agents won't get anywhere because any user process you want to automate is better done by creating APIs and creating a proper guaranteed interface. Any automated "computer use" will always be a one-off, absurdly expensive, and completely impractical.

I could see this coming into Apple Intelligence for example; you could simply ask the browser to buy stuff off of your favorite store, or even do a chain of tasks like informing a contact off your list that you've bought said thing, etc. The possibilities are quite exciting, in fact, even though the technology isn't quite there yet.

Apple should hook into app functions themselves instead of relying on UI. I would be really surprised if Apple made a browser automation tool, since that would be the complete opposite of the "it just works" credo

Re: Notes on Anthropic's Computer Use Ability

#82

Post by exe34 »

bamboozled wrote: post #59
exe34 wrote: post #57

Earlier quoted context omitted.

reality often is, unfortunately.

Depressing hasn’t been the reality for the majority of people over the last 100 years of technological progress. You could die from a scratch or a kidney stone 100s ago. Maybe this is the cliff , but it feels unlikely.

these things weren't depressing back then, they were normal. you can be depressed anywhere or anywhen, and similarly happy in any circumstances.

Re: Notes on Anthropic's Computer Use Ability

#83

Post by ctoth »

_heimdall wrote: post #18

I'm all for the MVP approach and shipping quickly, though I'm really surprised they went with image recognition and tooling for injecting mouse/keyboard events for automating human tasks. I wonder why leveraging accessibility tools for this wouldn't have been a better option. Browsers and operating systems both have pretty comprehensive tooling for accessibility tools like screen readers, and the whole point of those…

Hmmm.

I think I'm a blind user of the late 2024 Internet.

I use a screen reader for everything.

Now I'm paranoid that I'm just a computer use model, testing this a11y API hypothesis, in training.

Re: Notes on Anthropic's Computer Use Ability

#84

Post by pinko »

steveBK123 wrote: post #76
acrooks wrote: post #70

Earlier quoted context omitted.

While the strategy to find good deals / award space is a solved problem, the search tools to do so aren't. Tools like ExpertFlyer are super inefficient: it permits you to search for maximum one origin + one destination + one airline per search. What if you're happy to go to anywhere in Western Europe? Or if you want to check several different airlines? Then all of a sudden your one EF search might turn into dozens. A…

Yes I think this points to the need for adaptiveness which remains humans edge. We don't need PBs of training data, millions of compute, and hours upon hours of training. You could sit down a moderately intelligent intern as a mechanical turk to perform this workflow with only a few minutes of instruction and get a reasonably good result.

Underrated comment.

Re: Notes on Anthropic's Computer Use Ability

#85

Post by alpha_squared »

danielbln wrote: post #79
imranq wrote: post #73

This is basically RPA with LLMs. And RPA is basically the worst possible solution to any problem. Agents won't get anywhere because any user process you want to automate is better done by creating APIs and creating a proper guaranteed interface. Any automated "computer use" will always be a one-off, absurdly expensive, and completely impractical.

There is plenty of legacy software out there that has no and will never have a nice API to integrate with. Those are the situations where the terrible solutions are either let a human do it or automate the human tool chain from a high level. This is the LLM spin on it. Is it an efficient or even good solution? Hell no, but if there is no other solution to automation (assuming that's the goal) then does that matter?

This is a severely under-appreciated perspective. A lot of software, especially in industries that are slow to change, is just not programming-friendly. There are no APIs and no access to underlying databases, just user-focused point-and-click.

Re: Notes on Anthropic's Computer Use Ability

#86

Post by Facemelters »

bonoboTP wrote: post #68

This kind of stuff is an existential threat to ad-based business models and upselling. If users no longer browse the web themselves, you can't show them ads. It's a monumental, Earth-shattering problem for behemoth like Google but also normal websites. Lots of websites (such as booking.com) rely on shady practices to mislead users and upsell them etc. If you have a dispassionate, smart computer agent doing the transa…

the ads will target the latent biases of the agentic AI, just like they do with humans

Re: Notes on Anthropic's Computer Use Ability

#87

Post by lupusreal »

tracerbulletx wrote: post #71
bonoboTP wrote: post #68

This kind of stuff is an existential threat to ad-based business models and upselling. If users no longer browse the web themselves, you can't show them ads. It's a monumental, Earth-shattering problem for behemoth like Google but also normal websites. Lots of websites (such as booking.com) rely on shady practices to mislead users and upsell them etc. If you have a dispassionate, smart computer agent doing the transa…

Ads will move to the layer of the new interface when that happens. Also a computer can't watch a youtube video for you or look at funny cat pictures. You can still put ads next to things people want to look at.

Only a matter of time before computers are generating videos of cats cuter than any real cat.

Re: Notes on Anthropic's Computer Use Ability

#88

Post by pier25 »

nilstycho wrote: post #29

It seems like a cheaper intermediate capability would be to give Claude the ability to SSH to your computer or to a cloud container. That would unlock a lot of possibilities, without incurring the cost of the vision model or the difficulty of cursor manipulation. Does this already exist? If not, would the benefits be lower than I think, or would the costs be higher than I think?

I would never allow an AI to SSH into a server.

Just the other day someone used Claude to write a script to configure a server. It left a port open and the server was hacked hours later and used to attack other servers. Hetzner almost banned the hosting account.

https://x.com/rameerez/status/1848707234068382001

Re: Notes on Anthropic's Computer Use Ability

#89

Post by lupusreal »

ryukafalz wrote: post #51
infecto wrote: post #24

Earlier quoted context omitted.

Those sound like stop gaps at best. Its pretty clear the intended goal here. APIs are easy to integrate with but most systems in existence only have a visual interface intended for humans. The end goal here is clear, being able to interface with anything available in the screen.

Accessibility tools are made for humans. If there is information only available visually and not via a screen reader or other accessibility tools, that is a problem that needs to be addressed.

It's like making a robot that can walk up stairs instead of roll up wheelchair ramps or use elevators. It's harder, but more capable.
Post reply on HN