Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

anthropic.com

Post reply on HN
Print view View original

431–440 of 758 posts

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#431

Post by ianeigorndua »

anotherpaulg wrote: post #416

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

Are these synthetic or real-world benchmarks?

Answering myself: ”Aider’s code editing benchmark asks the LLM to edit python source files to complete 133 small coding exercises from Exercism”

Not gonna start looking for a job any time soon

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#432

Post by 93po »

jusgu wrote: post #382
bilsbie wrote: post #361

Does this make cursor obsolete? You can just use any IDE you want and it will work with it.

Assuming running this new computer interactivity feature is as fast as cursor composer (which I don’t think it is)—it still doesn’t support codebase indexing, inline edits or references to other variables and files in the codebase. I can see how someone could use this to make some sort of cursor competitor but out of the box there’s a very low likelihood it makes cursor obsolete.

i really want cursor to integrate this so it can look at the results of a code change in the browser and then make edits as needed until it's accomplished what i asked of it. same for errors in the console etc. right now i have to manually describe the issue or copy and paste the error message and it'd be nice for it to just iterate more on its own

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#433

Post by hawk_ »

aerhardt wrote: post #408
nopinsight wrote: post #376

Earlier quoted context omitted.

Self-driving is a beyond-six-sigma problem. An error rate of over 1-2 crashes per million miles, i.e., the human rate, is unacceptable. Most jobs are not like that. A good argument can be made, however, that software engineering, especially in important domains, will be among the last to be fully automated because software errors often cascade. There’s a countervailing effect though. It’s easy to generate and validat…

> software errors often cascade Whereas software defects in design and architecture subtly accumulate , until they leave the codebase in a state in which it becomes utterly unworkable. It is one of the chief reasons why good devs get paid what they do. Software discussions very often underrate software extensibility, or in other words, its structural and architectural scaleability. Even software correctness is trivia…

I hear you. But I have wondered if there won't be a need to maintain certain like of software when you can just have it be rewritten for each iteration. Like some kind of schema evolution, yes but throwaway software at each iteration.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#434

Post by sdwr »

LASR wrote: post #372
pants2 wrote: post #357

Earlier quoted context omitted.

Basically, if it means companies can introduce automation without changing anything about the tooling/workflow/programs they already use, it's going to be MASSIVE. Just and install and a prompt and you've already automated a lengthy manual process - awesome.

That's exactly it. I've been peddling my vision of "AI automation" for the last several months to acquaintances of mine in various professional fields. In some cases, even building up prototypes and real-user testing. Invariably, none have really stuck. This is not a technical problem that requires a technical solution. The problem is that it requires human behavior change. In the context of AI automation, the promis…

There's a huge huge gap between "coaxing what you want out of it" and "trusting it to perform flawlessly". Everybody on the planet would use #2, but #1 is just for enthusiasts.

AI is squarely #1. You can't trust it with your credit card to order groceries, or to budget and plan and book your vacation. People aren't picking up on AI because it isn't good enough yet to trust - you still have the burden of responsibility for the task.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#435

Post by mercacona »

I'm giving the new Sonnet a chance, although for my use as a writing companion so far, Opus has been king among all the models I've tried.

However, I've been using Opus as a writing companion for several months, especially when you have writer's block and ask it for alternative phrases, it was super creative. But in recent weeks I was noticing a degradation in quality. My impression is that the model was degrading. Could this be technically possible? Might it be some kind of programmed obsolescence to hype new models?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#436

Post by shombaboor »

TaylorAlexander wrote: post #216

And today I realized that despite it being an extremely common activity, we don’t really have a word for “using the computer” which is distinct from “computing”. It’s funny because AI models are always “using a computer” but now they can “use your computer.”

what is the impact to desktop programs ui if it's just being manipulated by AI?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#437

Post by wewtyflakes »

I wonder if OpenAI will fast follow; usually they're the ones to throw down the gauntlet. That being said, you can play around with OpenAI with a similar architecture of vision + agent + exec + loop using Donobu, though it is constrained to web browsers.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#438

Post by IncreasePosts »

unshavedyak wrote: post #364
dmartinez wrote: post #347

Earlier quoted context omitted.

Every time I see this argument made, there seems to be a level of complexity and/or operational cost above which people throw up their hands and say "well of course we can't do that". I feel like we will see that again here as well. It really is similar to the self-driving problem.

I feel pain for the people who will be employed to "prompt engineer" the behavior of these things. When they inevitably hallucinate some insane behavior a human will have to take blame for why it's not working.. and yea, that'll be fun to be on the receiving end of.

That won't even be a real job. How exactly will there be this complex intelligence that can solve all these real world problems, but can't handle some ambiguity in some inputs it is provided? Wouldn't the ultra smart AI just ask clarifying questions so that literally anyone can "prompt engineer"?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#439

Post by sfink »

jimbokun wrote: post #362
nickserv wrote: post #328

Earlier quoted context omitted.

Nah, it's the equivalent of seeing faces in static, or animals in clouds. Our brains are hardwired to see patterns, even when there are none. A similar, and related, behavior is seeing intent and intelligence in random phenomenon.

So it's behaving like our brains. Yet it's not AGI. Does that mean our brains do not implement General Intelligence?

When I take a dump, I do it exactly like a violin virtuoso does. I am not a violin virtuoso.

Trust me, I'm really not.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#440

Post by artemisart »

anotherpaulg wrote: post #416

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

Thanks! I was waiting for your benchmarks. Do you plan to test haiku 3.5 too? It would be nice to show API prices needed to run the whole benchmark too to have a better idea of how many internal tokens o1 models consume.
Post reply on HN