Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on Anthropic's Computer Use Ability

composio.dev

Post reply on HN
Print view View original

51–60 of 108 posts

Re: Notes on Anthropic's Computer Use Ability

#51

Post by ryukafalz »

infecto wrote: post #24
_heimdall wrote: post #18

I'm all for the MVP approach and shipping quickly, though I'm really surprised they went with image recognition and tooling for injecting mouse/keyboard events for automating human tasks. I wonder why leveraging accessibility tools for this wouldn't have been a better option. Browsers and operating systems both have pretty comprehensive tooling for accessibility tools like screen readers, and the whole point of those…

Those sound like stop gaps at best. Its pretty clear the intended goal here. APIs are easy to integrate with but most systems in existence only have a visual interface intended for humans. The end goal here is clear, being able to interface with anything available in the screen.

Accessibility tools are made for humans. If there is information only available visually and not via a screen reader or other accessibility tools, that is a problem that needs to be addressed.

Re: Notes on Anthropic's Computer Use Ability

#52

Post by steveBK123 »

im3w1l wrote: post #26
sunilkumardash9 wrote: post #12

Earlier quoted context omitted.

This is a step towards a human-machine hybrid world. Putting a human in the loop can do wonders. Sure, it is expensive now, but the subsequent iterations will crush it.

Have you heard of Centaur chess? A human and a machine would team up to find the best chess moves against another similar team. It's not a thing anymore. Computers have advanced so much that humans can't really contribute in any meaningful sense.

All these AI models do quite well in games because there are set rules, finite moves, and they can iterate in a tight loop (without humans) to get immediate feedback on pass/fail.

I think this is what differentiates the speed at which AIs have gotten from ok -> good -> great -> better than humans at say chess, versus say driving a car, summarizing a paper, understanding human requests, recommending music, etc.

I think a lot of people are extrapolating the rate of progress & possible accuracy rates from chess bots to domains that do not compare.

Re: Notes on Anthropic's Computer Use Ability

#53

Post by bamboozled »

bitwize wrote: post #36
blurbleblurble wrote: post #28

Earlier quoted context omitted.

This is an insightful comment, though it just goes to show how rigid the framing is of "natural vs. artificial" or "human vs. machine". None of this stuff has any vitality outside of _some_ relationship or interface with people.

Yeah, it makes the owner class richer while driving the marginal cost of labor to zero, at which point the working class can't sell their labor at all and starve.

This would assume the rich some how oppresses everyone to pieces. If I have access to all this wonderful automation tech, I'm sure as fuck not going to sit around and starve, I'm going to try automate my food production to make more food, more efficiently ?

Re: Notes on Anthropic's Computer Use Ability

#54

Post by Slump »

nerdix wrote: post #25
namanyayg wrote: post #8

What are some good use cases for this? Something that a business can be built around

It's main use case is making the average office worker feel the same existential dread that some programmers feel when they see a LLM spit out a bunch of code in mere seconds.

You're not far off. Anecdotal but I shared the Anthropic demo video and a few articles in a company slack and a lot of PM's/admin folks that are only tangentially aware of LLM powered use cases at this point shared that sentiment. Welcome to the party folks!

Re: Notes on Anthropic's Computer Use Ability

#55

Post by lagniappe »

nerdix wrote: post #25
namanyayg wrote: post #8

What are some good use cases for this? Something that a business can be built around

It's main use case is making the average office worker feel the same existential dread that some programmers feel when they see a LLM spit out a bunch of code in mere seconds.

I don't normally go for comedy on HN but this one got an audible chuckle.

TBH, while I giggle at the thought of anybody being replaced, I dont think it's likely, it's just that the standards and expectations have shifted in some domains. I think if anything LLM's raised the tide for everyone (in relevant roles) and we're all able to move a little faster now, like when we went from abacus to calculator a while back, just a different scale of magnitude.

Re: Notes on Anthropic's Computer Use Ability

#56

Post by steveBK123 »

belval wrote: post #49
steveBK123 wrote: post #41

Earlier quoted context omitted.

I've seen similar sentiment even pre-LLM that AI would help automate other forms of testing, and I just don't quite see it. Implementing tests is not the hard part. You could make that an intern project or hire a consultant for 3 months. The hard part is the interpretation of results. That is - making a thing that spits out tickets/alerts is easy. The signal/noise tuning and actual investigation workflows are the har…

> making a thing that spits out tickets/alerts is easy. I don't really believe that what I am asking for is hard, yet I still can't buy it as far as I know. > actual investigation workflows are the hard part and still very manual & human operated. Sure but it would allow your QA worker to have pre-tested usecase-based path with some flag on whether or not they may be problematic with a screen-recording and some times…

Fair - I'm not personally familiar with state of the art in UI QA automation, but I know theres been various screen recording type tools available for a decade+ with mixed success.

I come more from a "big data" background, and have dealt with CTOs who think "can't we just use AI?" is the answer to data quality checking multi-PB data lakes with 1000s of unique datasets from 100s of vendors. That is - they don't want to staff a data quality team, they think you can just magic it all away.

The answer was always - sure, but you are fixated on the easy part - anomaly detection. Actual data analysis on what broke, when, how, why, and escalating to data provider was always 95% of the work. Someone needs to look at the exhaust, and there will be exhaust every single day.. so you can kill your dev teams productivity or actually staff an operations team responsible for the tickets the thing spits out.

Re: Notes on Anthropic's Computer Use Ability

#57

Post by exe34 »

bamboozled wrote: post #47
exe34 wrote: post #34

Earlier quoted context omitted.

is this a cry for help? there's always alcohol and drugs, they can't take that away from us! (unless the robots of the future are like Bender)

I'm not saying I am depressed, but I mean, the comment just sounded like such a major downer.

reality often is, unfortunately.

Re: Notes on Anthropic's Computer Use Ability

#58

Post by steveBK123 »

acrooks wrote: post #15

I've built a couple of experiments using it so far and it has been really interesting. On one hand, it has really helped me with prototyping incredibly fast. On the other, it is prohibitively expensive today. Essentially you pay per click, in some cases per keystroke. I tried to get it to find a flight for me. So it opened the browser, navigated to Google Flights, entered the origin, destination etc. etc. By the time…

I guess I'm confused there's even a use case there. It's like "let me google that for you". I mean Siri can return me search results for flights.

A real killer app would be something that is adaptive and smart enough to deal with all the SEO/walled gardens in the travel search space, actually understanding the airlines available and searching directly there as well as at aggregators. It could also be integrated with your Airline miles accounts and all suggested options to use miles/miles&cash/cash, etc.

All of that is far more complex than .. clicking around google flights on your behalf and crashing.

Further, the real killer app is that it is bullet proof enough that you entrust it to book said best flight for you. This requires getting the product to 99.99% rather than the perpetual 70-80% we are seeing all these LLM use cases hit.

Re: Notes on Anthropic's Computer Use Ability

#59

Post by bamboozled »

exe34 wrote: post #57
bamboozled wrote: post #47

Earlier quoted context omitted.

I'm not saying I am depressed, but I mean, the comment just sounded like such a major downer.

reality often is, unfortunately.

Depressing hasn’t been the reality for the majority of people over the last 100 years of technological progress. You could die from a scratch or a kidney stone 100s ago.

Maybe this is the cliff , but it feels unlikely.

Re: Notes on Anthropic's Computer Use Ability

#60

Post by belval »

steveBK123 wrote: post #56
belval wrote: post #49

Earlier quoted context omitted.

> making a thing that spits out tickets/alerts is easy. I don't really believe that what I am asking for is hard, yet I still can't buy it as far as I know. > actual investigation workflows are the hard part and still very manual & human operated. Sure but it would allow your QA worker to have pre-tested usecase-based path with some flag on whether or not they may be problematic with a screen-recording and some times…

Fair - I'm not personally familiar with state of the art in UI QA automation, but I know theres been various screen recording type tools available for a decade+ with mixed success. I come more from a "big data" background, and have dealt with CTOs who think "can't we just use AI?" is the answer to data quality checking multi-PB data lakes with 1000s of unique datasets from 100s of vendors. That is - they don't want t…

That's fair and I don't think I have a good counter to this, it would be very easy for such a UI QA product to become just another "security vulnerability scanner" that cuts low severity tickets that nobody looks at.
Post reply on HN