Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

anthropic.com

Post reply on HN
Print view View original

491–500 of 758 posts

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#491

Post by stavros »

ianeigorndua wrote: post #431
anotherpaulg wrote: post #416

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

Are these synthetic or real-world benchmarks? Answering myself: ”Aider’s code editing benchmark asks the LLM to edit python source files to complete 133 small coding exercises from Exercism” Not gonna start looking for a job any time soon

I use Claude for coding and it's fantastic. I definitely have outsourced a lot of my coding to it.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#492

Post by falcor84 »

xxpor wrote: post #472
CSMastermind wrote: post #415

Earlier quoted context omitted.

RPA has been a huge pain to work with. It's almost always a framework around existing tools like Selenium that you constantly have to fight against to get good results from. I was always left with the feeling that I could build something better myself just handrolling the scripts rather than using their frameworks. Getting Claude integrated into the space is going to be a game changer.

I can see it now, Claude generating expect scripts. 1994 and 2024 will be fully joined.

The big thing I expect at the next level is in using Claude to first generate UI-based automation based on an end user's instructions, then automatically defining a suite of end-to-end tests, confirming with the user "is this how it should work?", and then finally using this suite to reimplement the flow from first principles.

I know we're still a bit far from there, but I don't see a particular hurdle that strikes me as requiring novel research.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#493

Post by Aeolun »

LASR wrote: post #372
pants2 wrote: post #357

Earlier quoted context omitted.

Basically, if it means companies can introduce automation without changing anything about the tooling/workflow/programs they already use, it's going to be MASSIVE. Just and install and a prompt and you've already automated a lengthy manual process - awesome.

That's exactly it. I've been peddling my vision of "AI automation" for the last several months to acquaintances of mine in various professional fields. In some cases, even building up prototypes and real-user testing. Invariably, none have really stuck. This is not a technical problem that requires a technical solution. The problem is that it requires human behavior change. In the context of AI automation, the promis…

There’s nothing to gain for anyone there. Workers will lose their jobs, and managers will lose their reports.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#494

Post by logankeenan »

simonw wrote: post #319

I wrote up some of my own notes on Computer Use here: https://simonwillison.net/2024/Oct/22/computer-use/

Molmo released recently and is able to provide point coordinates for objects in images. I’ve been testing it out recently and am currently building an automation tool that allows users to more easily control a computer. Looks like Anthropic built a better one.

Edit: it seems like these new features will eliminate a lot of automated testing tools we have today.

Code for molmo coordinate tests https://github.com/logankeenan/molmo-server

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#495

Post by vipshek »

joshuamcginnis wrote: post #469

Is there anything out there yet that will let me issue the command: > Refactor the api folder with any recommended readability improvements or improvements that would help DRY up code without adding additional complexity. Then I can just `git status` to see the changes?

Install Cursor (https://cursor.com), go into Cursor Settings and disable everything but Claude, then open Composer (Ctrl/Cmd + I). Paste in your exact command above. I bet it’ll do something pretty close to what you’re looking for.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#496

Post by falcor84 »

mewpmewp2 wrote: post #464
claytongulick wrote: post #459

Earlier quoted context omitted.

Sometimes. In my case I have a bunch of nurses that waste a huge amount of time dealing with clerical work and tech hoops, rather than operating at the top of their license. Traditional RPAs are tough when you're dealing with VPNs, 2fa, remote desktop (in multiple ways), a variety of EHRs and scraping clinical documentation from poorly structured clinical notes or PDFs. This technology looks like it could be a game c…

True, 2FA and all these little details that exist now have made this automation quite insanely complicated. It is of course necessary that we have 2FA etc, but there is huge potential in solving this I believe.

From a security standpoint, what's considered the "proper" way of assigning a bot access based on a person's 2FA? Would that be some sort of limited scope expiring token like GitHub's fine-grained personal access tokens?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#497

Post by stavros »

dimitri-vs wrote: post #479
claytongulick wrote: post #458

Earlier quoted context omitted.

It is. CTO of healthcare org here. I just put a hold on a new RPA project to keep an eye on this and see how it develops. According to their docs, Anthropic will sign a BAA.

Out of curiosity, how are high risk liability enviroments like yours coming to terms with the non-deterministic nature of models like these? Eg. the non-zero chance that it might click a button it *really* shouldn't as demonstrated in the failure demo.

Technical director at another company here: We have humans double-check everything, because we're required by law to. We use automation to make response times faster, or to do the bulk of the work and then just have humans double-check the AI. To do otherwise would be classed as "a software medical device", which needs documentation out the wazoo, and for good reason. I'm not sure you could even have a medical device where most of your design doc is "well I just hope it does the right thing, I guess?".

Sometimes, the AI is more accurate or safer than humans, but it still reads better to say "we always have humans in the loop". In those cases, we reap the benefits of both: Use the AI for safety, but still have a human fallback.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#498

Post by falcor84 »

quantadev wrote: post #199
gzer0 wrote: post #182

One of the funnier things during training with the new API (which can control your computer) was this: "Even while recording these demos, we encountered some amusing moments. In one, Claude accidentally stopped a long-running screen recording, causing all footage to be lost. Later, Claude took a break from our coding demo and began to peruse photos of Yellowstone National Park." [0] https://x.com/AnthropicAI/status/1…

I think the best use case for AI `Computer Use` would be a simple positioning of the mouse and asking for conformation before a click. For most use cases this is all people will want/need. If you don't know how to do something, it is basically teaching you how, in this case, rather than taking full control and doing things so fast you don't have time to stop of going rogue.

Even better, how about giving the AI the capability to move and draw and overlay on the screen with a separate virtual cursor as in a Zoom session?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#499

Post by simonw »

Claude 3.5 Opus is no longer mentioned at all on https://docs.anthropic.com/en/docs/about-claude/models

Internet Archive confirms that on the 8th of October that page listed 3.5 Opus as coming "Later this year" https://web.archive.org/web/20241008222204/https://docs.anth...

The fact that it's no longer listed suggests that its release has at least been delayed for an unpredictable amount of time, or maybe even cancelled.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#500

Post by falcor84 »

botro wrote: post #462
nomel wrote: post #273

Earlier quoted context omitted.

Not at all! Programs, and websites, are built for humans , and very very rarely offer non-GUI access. This is the only feasible way to make something useful now. I think it's also the reason why robots will look like humans, be the same proportions as humans, have roughly the same feet and hands as humans: everything in the world was designed for humans. That being the foundation is going to influence what's built on…

And to take a historic analogy, cars today are as wide as they are because that's about how wide a single lane roadway is. And a single lane roadway is as wide as it is because that's about the width of two horses drawing a carriage.

The story goes that this two horses width also limited the size of the space shuttle's boosters (SRB), so we ended up taking this sort of path-dependence off to space.
Post reply on HN