Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on Anthropic's Computer Use Ability

composio.dev

Post reply on HN
Print view View original

71–80 of 108 posts

Re: Notes on Anthropic's Computer Use Ability

#71

Post by tracerbulletx »

bonoboTP wrote: post #68

This kind of stuff is an existential threat to ad-based business models and upselling. If users no longer browse the web themselves, you can't show them ads. It's a monumental, Earth-shattering problem for behemoth like Google but also normal websites. Lots of websites (such as booking.com) rely on shady practices to mislead users and upsell them etc. If you have a dispassionate, smart computer agent doing the transa…

Ads will move to the layer of the new interface when that happens. Also a computer can't watch a youtube video for you or look at funny cat pictures. You can still put ads next to things people want to look at.

Re: Notes on Anthropic's Computer Use Ability

#72

Post by kredd »

bonoboTP wrote: post #68

This kind of stuff is an existential threat to ad-based business models and upselling. If users no longer browse the web themselves, you can't show them ads. It's a monumental, Earth-shattering problem for behemoth like Google but also normal websites. Lots of websites (such as booking.com) rely on shady practices to mislead users and upsell them etc. If you have a dispassionate, smart computer agent doing the transa…

Still not a problem for Meta/TikTok/YouTube though, as people go there to consume content on purpose. But I agree, will be fun to see how Google and others will deal with it.

Re: Notes on Anthropic's Computer Use Ability

#73

Post by imranq »

This is basically RPA with LLMs. And RPA is basically the worst possible solution to any problem.

Agents won't get anywhere because any user process you want to automate is better done by creating APIs and creating a proper guaranteed interface. Any automated "computer use" will always be a one-off, absurdly expensive, and completely impractical.

Re: Notes on Anthropic's Computer Use Ability

#74

Post by kordlessagain »

acrooks wrote: post #15

I've built a couple of experiments using it so far and it has been really interesting. On one hand, it has really helped me with prototyping incredibly fast. On the other, it is prohibitively expensive today. Essentially you pay per click, in some cases per keystroke. I tried to get it to find a flight for me. So it opened the browser, navigated to Google Flights, entered the origin, destination etc. etc. By the time…

It’s running in the browser but connected to a VM, right? When you say crashed, what did it do?

Re: Notes on Anthropic's Computer Use Ability

#75

Post by kordlessagain »

nilstycho wrote: post #29

It seems like a cheaper intermediate capability would be to give Claude the ability to SSH to your computer or to a cloud container. That would unlock a lot of possibilities, without incurring the cost of the vision model or the difficulty of cursor manipulation. Does this already exist? If not, would the benefits be lower than I think, or would the costs be higher than I think?

I’m working on Webwright which presents as a shell. It’s on GitHub.

Re: Notes on Anthropic's Computer Use Ability

#76

Post by steveBK123 »

acrooks wrote: post #70
sithadmin wrote: post #62

Earlier quoted context omitted.

The airline booking + awards redemption use case is a mostly solved problem. Harcore milage redemption enthusiasts use paid tools like ExpertFlyer that present a UI and API for peeking into airline reservation backends. It has a steep learning curve, for sure. ThePointsGuy blog tried to implement something that directly tied into airline accounts to track milage/points and redemption options, but I believe they got s…

While the strategy to find good deals / award space is a solved problem, the search tools to do so aren't. Tools like ExpertFlyer are super inefficient: it permits you to search for maximum one origin + one destination + one airline per search. What if you're happy to go to anywhere in Western Europe? Or if you want to check several different airlines? Then all of a sudden your one EF search might turn into dozens. A…

Yes I think this points to the need for adaptiveness which remains humans edge.

We don't need PBs of training data, millions of compute, and hours upon hours of training.

You could sit down a moderately intelligent intern as a mechanical turk to perform this workflow with only a few minutes of instruction and get a reasonably good result.

Re: Notes on Anthropic's Computer Use Ability

#77

Post by supriyo-biswas »

imranq wrote: post #73

This is basically RPA with LLMs. And RPA is basically the worst possible solution to any problem. Agents won't get anywhere because any user process you want to automate is better done by creating APIs and creating a proper guaranteed interface. Any automated "computer use" will always be a one-off, absurdly expensive, and completely impractical.

I could see this coming into Apple Intelligence for example; you could simply ask the browser to buy stuff off of your favorite store, or even do a chain of tasks like informing a contact off your list that you've bought said thing, etc.

The possibilities are quite exciting, in fact, even though the technology isn't quite there yet.

Re: Notes on Anthropic's Computer Use Ability

#78

Post by [deleted] »

imranq wrote: post #73

This is basically RPA with LLMs. And RPA is basically the worst possible solution to any problem. Agents won't get anywhere because any user process you want to automate is better done by creating APIs and creating a proper guaranteed interface. Any automated "computer use" will always be a one-off, absurdly expensive, and completely impractical.

[deleted]

Re: Notes on Anthropic's Computer Use Ability

#79

Post by danielbln »

imranq wrote: post #73

This is basically RPA with LLMs. And RPA is basically the worst possible solution to any problem. Agents won't get anywhere because any user process you want to automate is better done by creating APIs and creating a proper guaranteed interface. Any automated "computer use" will always be a one-off, absurdly expensive, and completely impractical.

There is plenty of legacy software out there that has no and will never have a nice API to integrate with. Those are the situations where the terrible solutions are either let a human do it or automate the human tool chain from a high level. This is the LLM spin on it. Is it an efficient or even good solution? Hell no, but if there is no other solution to automation (assuming that's the goal) then does that matter?

Re: Notes on Anthropic's Computer Use Ability

#80

Post by imranq »

danielbln wrote: post #79
imranq wrote: post #73

This is basically RPA with LLMs. And RPA is basically the worst possible solution to any problem. Agents won't get anywhere because any user process you want to automate is better done by creating APIs and creating a proper guaranteed interface. Any automated "computer use" will always be a one-off, absurdly expensive, and completely impractical.

There is plenty of legacy software out there that has no and will never have a nice API to integrate with. Those are the situations where the terrible solutions are either let a human do it or automate the human tool chain from a high level. This is the LLM spin on it. Is it an efficient or even good solution? Hell no, but if there is no other solution to automation (assuming that's the goal) then does that matter?

I'd argue that this is not even a solution to begin with. If the LLM gets even one pixel value wrong, then at best the whole process breaks down. At worst, you could do some irreversible damage.
Post reply on HN