Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on Anthropic's Computer Use Ability

composio.dev

Post reply on HN
Print view View original

101–108 of 108 posts

Re: Notes on Anthropic's Computer Use Ability

#101

Post by nilstycho »

kordlessagain wrote: post #75
nilstycho wrote: post #29

It seems like a cheaper intermediate capability would be to give Claude the ability to SSH to your computer or to a cloud container. That would unlock a lot of possibilities, without incurring the cost of the vision model or the difficulty of cursor manipulation. Does this already exist? If not, would the benefits be lower than I think, or would the costs be higher than I think?

I’m working on Webwright which presents as a shell. It’s on GitHub.

Based on the flow diagram, that doesn't seem to be the same thing. Webwright seems to be a shell as a tool for me, enhanced with AI features. I'm suggesting the shell as a tool for AI.

Webwright is a front-end shell that presents to me; I'm suggesting a back-end shell that presents to Claude.

It doesn't appear that Webwright enables tool-use. In other words, there's no task-oriented feedback loop between AI-provided shell commands and the results of those shell commands. Please correct me if that's not right.

Re: Notes on Anthropic's Computer Use Ability

#102

Post by jaggs »

steveBK123 wrote: post #76
acrooks wrote: post #70

Earlier quoted context omitted.

While the strategy to find good deals / award space is a solved problem, the search tools to do so aren't. Tools like ExpertFlyer are super inefficient: it permits you to search for maximum one origin + one destination + one airline per search. What if you're happy to go to anywhere in Western Europe? Or if you want to check several different airlines? Then all of a sudden your one EF search might turn into dozens. A…

Yes I think this points to the need for adaptiveness which remains humans edge. We don't need PBs of training data, millions of compute, and hours upon hours of training. You could sit down a moderately intelligent intern as a mechanical turk to perform this workflow with only a few minutes of instruction and get a reasonably good result.

Ah, but I think you're overlooking one major factor. Convenience. A lot of the spontaneous stuff we do ("hey why don't we pop down to x tomorrow?", or "do you fancy a quick curry?") are things you're not going to book with a Turk. BUT you definitely would fire up a quick agent on your way to the shower and have it do all the work for you while you're waxing your armpits. :) Agentic work is starting super slow, but once the wrinkles are worked out, we'll see a world where they're doing a huge amount of heavy lifting for the drudge stuff. For an example see Her - sorry! :)

Re: Notes on Anthropic's Computer Use Ability

#103

Post by pksebben »

steveBK123 wrote: post #52
im3w1l wrote: post #26

Earlier quoted context omitted.

Have you heard of Centaur chess? A human and a machine would team up to find the best chess moves against another similar team. It's not a thing anymore. Computers have advanced so much that humans can't really contribute in any meaningful sense.

All these AI models do quite well in games because there are set rules, finite moves, and they can iterate in a tight loop (without humans) to get immediate feedback on pass/fail. I think this is what differentiates the speed at which AIs have gotten from ok -> good -> great -> better than humans at say chess, versus say driving a car, summarizing a paper, understanding human requests, recommending music, etc. I thin…

Would've said that about writing and text, about three years ago.

Re: Notes on Anthropic's Computer Use Ability

#104

Post by _heimdall »

elif wrote: post #44
_heimdall wrote: post #18

I'm all for the MVP approach and shipping quickly, though I'm really surprised they went with image recognition and tooling for injecting mouse/keyboard events for automating human tasks. I wonder why leveraging accessibility tools for this wouldn't have been a better option. Browsers and operating systems both have pretty comprehensive tooling for accessibility tools like screen readers, and the whole point of those…

Crazy that this needs to be said but 'Computer use' is far more expansive of a domain than Internet browsing...

From what I've seen of this new product (I've never used it), it sounds like its specifically trying to mimic a human user and they went with image recognition plus faked input devices.

That approach is a weird one to me, though only as long as its limited to the current use. If this is just another test bed for a much more broad tool that could rely on accessibility APIs that makes sense.

Re: Notes on Anthropic's Computer Use Ability

#105

Post by _heimdall »

danielbln wrote: post #22
_heimdall wrote: post #18

I'm all for the MVP approach and shipping quickly, though I'm really surprised they went with image recognition and tooling for injecting mouse/keyboard events for automating human tasks. I wonder why leveraging accessibility tools for this wouldn't have been a better option. Browsers and operating systems both have pretty comprehensive tooling for accessibility tools like screen readers, and the whole point of those…

I think the reason is that this is the most general implementation. It doesn't need playwright or have access to the DOM or anything else, if it has a screen and mouse/keyboard, then it will work. That's quite powerful (if slow and pricey, at the moment).

Unless I'm mistaken, playwright doesn't actually use the accessibility tree directly. It does have quite a few APIs for accessing nodes based on a11y attributes, but I could have sworn those were glorified query selectors rather than directly accessing the accessibility tree.

Last I checked on it, maybe a year ago, there were browser proposals for standardizing the accessibility tree APIs but they were very early discussions and seemed pretty well stuck.

That would be a good reason for Anthropic using image processing here though, short of forking open source a11y tools there may not have been a simple way to use accessibility data to interact.

Re: Notes on Anthropic's Computer Use Ability

#106

Post by tsunamifury »

bamboozled wrote: post #45
tsunamifury wrote: post #37

Earlier quoted context omitted.

Once we realize we can make machines that can beat us in ways we can’t even understand, I wonder if will question if we have always been influenced this way by an exterior force

Sounds like an interesting idea, do you mean, like the concept of "fate" is the type of external force you describe ?

Yea something like that or like the benejeserit from dune.

Re: Notes on Anthropic's Computer Use Ability

#107

Post by nashadelic »

steveBK123 wrote: post #41
belval wrote: post #35

The product I would like to see out of this is a way to automate UI QA. Ideally it would be given a persona and a list of use cases, try to accomplish each task and save the state where you/it failed. Something like a Chrome lighthouse but for usability. Bonus point if it can highlight what part of my documentation is using mismatched terminology making it difficult for newcomers to understand what button I am referr…

I've seen similar sentiment even pre-LLM that AI would help automate other forms of testing, and I just don't quite see it. Implementing tests is not the hard part. You could make that an intern project or hire a consultant for 3 months. The hard part is the interpretation of results. That is - making a thing that spits out tickets/alerts is easy. The signal/noise tuning and actual investigation workflows are the har…

It’s both. Most manual tests are required to be run whenever the underlying code has changed. And that’s pretty slow and annoying. Interpreting results is usually pretty trivial, like checking the http code or checking against an assert. I don’t think most companies use/should use manual testing but where it’s unavoidable, this is a great workaround.

Re: Notes on Anthropic's Computer Use Ability

#108

Post by dhorthy »

belval wrote: post #60
steveBK123 wrote: post #56

Earlier quoted context omitted.

Fair - I'm not personally familiar with state of the art in UI QA automation, but I know theres been various screen recording type tools available for a decade+ with mixed success. I come more from a "big data" background, and have dealt with CTOs who think "can't we just use AI?" is the answer to data quality checking multi-PB data lakes with 1000s of unique datasets from 100s of vendors. That is - they don't want t…

That's fair and I don't think I have a good counter to this, it would be very easy for such a UI QA product to become just another "security vulnerability scanner" that cuts low severity tickets that nobody looks at.

do y'all see a way to ramp from mostly-human-in-the-loop to mostly-ai? Can you take a system that does 1% at the hard part of signal/tuning and teach it to get better over time?

I'm thinking for a single particular application under test and a mostly-static group of SMEs who might be involved to respond/tune

Post reply on HN