Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on Anthropic's Computer Use Ability

composio.dev

Post reply on HN
Print view View original

61–70 of 108 posts

Re: Notes on Anthropic's Computer Use Ability

#61

Post by computerthings »

bamboozled wrote: post #53
bitwize wrote: post #36

Earlier quoted context omitted.

Yeah, it makes the owner class richer while driving the marginal cost of labor to zero, at which point the working class can't sell their labor at all and starve.

This would assume the rich some how oppresses everyone to pieces. If I have access to all this wonderful automation tech, I'm sure as fuck not going to sit around and starve, I'm going to try automate my food production to make more food, more efficiently ?

> If I have access to all this wonderful automation tech

But "you" don't, that is precisely the point. The speed at which the gap between rich and poor grows keeps increasing, after all -- the rest is commentary --, and people who right now send people to die and murder in wars for oil, and what not not, will not suddenly start sharing when they fully captured all means of production for good. That's like hoping the person who keeps stealing your shit every chance he has, leaving you in sickness or death without a thought will give you a billion dollars once all the locks on your house have rusted off completely and you no longer have means to call the police.

Re: Notes on Anthropic's Computer Use Ability

#62

Post by sithadmin »

steveBK123 wrote: post #58
acrooks wrote: post #15

I've built a couple of experiments using it so far and it has been really interesting. On one hand, it has really helped me with prototyping incredibly fast. On the other, it is prohibitively expensive today. Essentially you pay per click, in some cases per keystroke. I tried to get it to find a flight for me. So it opened the browser, navigated to Google Flights, entered the origin, destination etc. etc. By the time…

I guess I'm confused there's even a use case there. It's like "let me google that for you". I mean Siri can return me search results for flights. A real killer app would be something that is adaptive and smart enough to deal with all the SEO/walled gardens in the travel search space, actually understanding the airlines available and searching directly there as well as at aggregators. It could also be integrated with…

The airline booking + awards redemption use case is a mostly solved problem. Harcore milage redemption enthusiasts use paid tools like ExpertFlyer that present a UI and API for peeking into airline reservation backends. It has a steep learning curve, for sure.

ThePointsGuy blog tried to implement something that directly tied into airline accounts to track milage/points and redemption options, but I believe they got slapped down by several airlines for unauthorized scraping. Airlines do NOT like third parties having access to frequent flier accounts.

Re: Notes on Anthropic's Computer Use Ability

#63

Post by steveBK123 »

sithadmin wrote: post #62
steveBK123 wrote: post #58

Earlier quoted context omitted.

I guess I'm confused there's even a use case there. It's like "let me google that for you". I mean Siri can return me search results for flights. A real killer app would be something that is adaptive and smart enough to deal with all the SEO/walled gardens in the travel search space, actually understanding the airlines available and searching directly there as well as at aggregators. It could also be integrated with…

The airline booking + awards redemption use case is a mostly solved problem. Harcore milage redemption enthusiasts use paid tools like ExpertFlyer that present a UI and API for peeking into airline reservation backends. It has a steep learning curve, for sure. ThePointsGuy blog tried to implement something that directly tied into airline accounts to track milage/points and redemption options, but I believe they got s…

Yes, that seems to be the larger challenge. The search tools I have used will work for a while until they don't. Real cat & mouse game.

Hence the "adaptive" part of my comment.

It really needs to be a client side agent.

Re: Notes on Anthropic's Computer Use Ability

#64

Post by infecto »

ryukafalz wrote: post #51
infecto wrote: post #24

Earlier quoted context omitted.

Those sound like stop gaps at best. Its pretty clear the intended goal here. APIs are easy to integrate with but most systems in existence only have a visual interface intended for humans. The end goal here is clear, being able to interface with anything available in the screen.

Accessibility tools are made for humans. If there is information only available visually and not via a screen reader or other accessibility tools, that is a problem that needs to be addressed.

Accessibility tools are I find are never as great as the source. Just because they are made for humans does not mean they are an improvement. I imagine at best a stopgap as image models improve.

Re: Notes on Anthropic's Computer Use Ability

#65

Post by Jayakumark »

Any idea on how does Sonnet does this, is the image annotated with bounding boxes on text boxes etc. along with its coordinates before sending to sonnet and it responds with box name back or co-ordinate back or ? is SAM2 used for segmenting everything before sending to sonnet ?

Re: Notes on Anthropic's Computer Use Ability

#67

Post by azinman2 »

I saw those demoed yesterday. The model was asked to create a cool visualization. It ultimately tried to install steamlit and go its page, only to find its own Claude software running streamlit, so as part of debugging it killed itself. Not ready to let that go wild on my own computer!

Re: Notes on Anthropic's Computer Use Ability

#68

Post by bonoboTP »

This kind of stuff is an existential threat to ad-based business models and upselling. If users no longer browse the web themselves, you can't show them ads. It's a monumental, Earth-shattering problem for behemoth like Google but also normal websites. Lots of websites (such as booking.com) rely on shady practices to mislead users and upsell them etc. If you have a dispassionate, smart computer agent doing the transaction, it will only buy what's needed to accomplish the task.

There will be enormous push towards steering these software agents towards similarly shady practices instead of making them act in the true interest of the user. The ads will be built into the weights of the model or something.

Re: Notes on Anthropic's Computer Use Ability

#70

Post by acrooks »

sithadmin wrote: post #62
steveBK123 wrote: post #58

Earlier quoted context omitted.

I guess I'm confused there's even a use case there. It's like "let me google that for you". I mean Siri can return me search results for flights. A real killer app would be something that is adaptive and smart enough to deal with all the SEO/walled gardens in the travel search space, actually understanding the airlines available and searching directly there as well as at aggregators. It could also be integrated with…

The airline booking + awards redemption use case is a mostly solved problem. Harcore milage redemption enthusiasts use paid tools like ExpertFlyer that present a UI and API for peeking into airline reservation backends. It has a steep learning curve, for sure. ThePointsGuy blog tried to implement something that directly tied into airline accounts to track milage/points and redemption options, but I believe they got s…

While the strategy to find good deals / award space is a solved problem, the search tools to do so aren't. Tools like ExpertFlyer are super inefficient: it permits you to search for maximum one origin + one destination + one airline per search. What if you're happy to go to anywhere in Western Europe? Or if you want to check several different airlines? Then all of a sudden your one EF search might turn into dozens. And as you say, pretty much all of the aggregator tools are getting slapped down by airlines so they increasingly have more limited availability and some are shutting down completely.

And then add the complexity that you might be willing to pay cash if the price is right ... so then you add dozens more searches to that on potentially many websites.

All of this is "easy" and a solved problem but it's incredibly monotonous. And almost none of these services offer an API, so it's difficult to automate without a browser-based approach. And a travel agent won't work this hard for you. So how amazing would it be instead to tell an AI agent what you want, have it pretend to be you for a few minutes, and get a CSV file in your inbox at the end.

Whether this could be commercialised is a different question but I'm certainly going to continue building out my prototype to save myself some time (I mean, to be fair, it will probably take more time to build something to do this on my behalf but I think it's time well spent).

Post reply on HN