Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

anthropic.com

Post reply on HN
Print view View original

571–580 of 758 posts

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#571

Post by anonzzzies »

faizshah wrote: post #501
anotherpaulg wrote: post #416

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

Anecdotally but I still get significantly better results from ChatGPT than claude for coding. Claude is way less controllable it is difficult to get it to do exactly what I want. ChatGPT is way easier to control in terms of asking for specific changes. Not sure why that is maybe the chain of thought and instruction tuning dataset has made theirs a lot better for interactive use.

For me it's the opposite; chatgpt (o1 preview and 4o) keep making very strange errors; errors that I even exactly tell it how to fix and it simply repeats the fundamental mistakes again. With claude, I did not have that.

Example; I asked it to write some js that finds a button on a page, clicks the button, then waits for a new element with some selector to appear and return a ref to it; chatgpt kept returning (pseudo code);

while (true) {

button.click()

wait()

oldItems = ...

newItems = ...

newItem = newItems - oldItems

if (newItem) return newItem

sleep(1)

}

which obviously doesn't work. Claude understands to put the oldItems outside the while; even when I tell chatgpt to do that, it doesn't. Or it does one time and with another change, it moves it back in.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#572

Post by quantadev »

falcor84 wrote: post #498
quantadev wrote: post #199

Earlier quoted context omitted.

I think the best use case for AI `Computer Use` would be a simple positioning of the mouse and asking for conformation before a click. For most use cases this is all people will want/need. If you don't know how to do something, it is basically teaching you how, in this case, rather than taking full control and doing things so fast you don't have time to stop of going rogue.

Even better, how about giving the AI the capability to move and draw and overlay on the screen with a separate virtual cursor as in a Zoom session?

I like the drawing on the screen idea. The biggest use case of that I can think of is drawing a black rectangle over all ADs!!!!

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#573

Post by p1esk »

ianeigorndua wrote: post #568
stavros wrote: post #491

Earlier quoted context omitted.

I use Claude for coding and it's fantastic. I definitely have outsourced a lot of my coding to it.

Sure it can do coding but can it do software engineering

What exactly is left when we remove coding from software engineering? Could it be handled by a manager? Or perhaps by a single senior SWE who could now perform the work of an entire team using these rapidly advancing AI coders?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#574

Post by findjashua »

p1esk wrote: post #573
ianeigorndua wrote: post #568

Earlier quoted context omitted.

Sure it can do coding but can it do software engineering

What exactly is left when we remove coding from software engineering? Could it be handled by a manager? Or perhaps by a single senior SWE who could now perform the work of an entire team using these rapidly advancing AI coders?

for a lot of tasks that aren't as cut & dry, i often find myself having to provide it pseudo code, which it can then one-shot to working code.

don't get me wrong, it's still a massive upgrade from the pre-sonnet era, but i still don't think it can take a high-level requirement and convert it into a working project... yet

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#575

Post by mirsadm »

nopinsight wrote: post #339

This needs more discussion: Claude using Claude on a computer for coding https://youtu.be/vH2f7cjXjKI?si=Tw7rBPGsavzb-LNo (3 mins) True end-user programming and product manager programming are coming, probably pretty soon. Not the same thing, but Midjourney went from v.1 to v.6 in less than 2 years. If something similar happens, most jobs that could be done remotely will be automatable in a few years.

It makes me wonder if people that make these claims have an actual job. Because if they did then I doubt anyone could make that claim with a straight face.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#576

Post by swyx »

akshayKMR wrote: post #211
swyx wrote: post #99

my quick notes on Computer Use: - "computer use" is basically using Claude's vision + tool use capability in a loop. There's a reference impl but there's no "claude desktop" app that just comes with this OOTB - they're basically advertising that they bumped up Claude 3.5's screen vision capability. we discussed the importance of this general computer agent approach with David on our pod https://x.com/swyx/status/1771…

Haven't used vision models before, can someone comment if they are good at "pointing things". E.g given a picture, give co-ordinate for text "foo". This is the key to accurate control, it needs to be very precise. Maybe Claude's model is trained at this. Also what about open source vision models? Any ones good at "pointing things" on a typical computer screen?

i mean like with everything they'll kinda be able to do it and only get really good at it if the model trainers prioritized it. see Pixmo for a recent example

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#577

Post by itchyjunk »

lobochrome wrote: post #567
csomar wrote: post #561

I like its lazy approach > I apologize, but it seems we're having some issues installing Chromium. As an alternative, since I can see Firefox is already installed on the system, would you like to use Firefox instead? Or would you prefer me to try a different approach to installing Chromium? It seems unable to figure out how to install chormium. Sees the Firefox icon out there and just want to use it. You know, like a…

SYSTEM_PROMPT = f""" * You are utilising an Ubuntu virtual machine using {platform.machine()} architecture with internet access. * You can feel free to install Ubuntu applications with your bash tool. Use curl instead of wget. * To open firefox, please just click on the firefox icon. Note, firefox-esr is what is installed on your system. * Using bash tool you can start GUI applications, but you need to set export DIS…

Is this you guessing the system prompt? Is this the actual prompt the model has? What is the context?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#578

Post by gorgoiler »

ianeigorndua wrote: post #568
stavros wrote: post #491

Earlier quoted context omitted.

I use Claude for coding and it's fantastic. I definitely have outsourced a lot of my coding to it.

Sure it can do coding but can it do software engineering

It’s a good question. I would ask…

(1) Sure, it can tell you how to write new code in response to a prompt about your current local problem, but

(2) can it reason about an entire code base of known and unknown problems, and use that basis to figure out solutions to the unknowns such that you delete code and collapse complexity.

The software equivalent of realising that if you subtract xy from this:

  x2 + 3xy + y2
You can turn it into a much neater version:

  (x + y)2 + xy
…but doing that with 100k tokens of code instead of a handful of algebra tokens.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#579

Post by lukasb »

ianeigorndua wrote: post #568
stavros wrote: post #491

Earlier quoted context omitted.

I use Claude for coding and it's fantastic. I definitely have outsourced a lot of my coding to it.

Sure it can do coding but can it do software engineering

I haven't had much luck with architecture stuff. Maybe I'm holding it wrong.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#580

Post by seriocomic »

stavros wrote: post #491
ianeigorndua wrote: post #431

Earlier quoted context omitted.

Are these synthetic or real-world benchmarks? Answering myself: ”Aider’s code editing benchmark asks the LLM to edit python source files to complete 133 small coding exercises from Exercism” Not gonna start looking for a job any time soon

I use Claude for coding and it's fantastic. I definitely have outsourced a lot of my coding to it.

What's the (current) best way to integrate it? VS Code extension? Other IDE?
Post reply on HN