Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

anthropic.com

Post reply on HN
Print view View original

501–510 of 758 posts

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#501

Post by faizshah »

anotherpaulg wrote: post #416

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

Anecdotally but I still get significantly better results from ChatGPT than claude for coding.

Claude is way less controllable it is difficult to get it to do exactly what I want. ChatGPT is way easier to control in terms of asking for specific changes.

Not sure why that is maybe the chain of thought and instruction tuning dataset has made theirs a lot better for interactive use.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#502

Post by falcor84 »

joshuamcginnis wrote: post #469

Is there anything out there yet that will let me issue the command: > Refactor the api folder with any recommended readability improvements or improvements that would help DRY up code without adding additional complexity. Then I can just `git status` to see the changes?

Aider is great at this stuff. The recommended way is to have it automatically commit, and then you can examine and possibly revert/reset its commits (or just have it work on a separate branch), but you can also use --no-auto-commits

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#503

Post by tkgally »

famahar wrote: post #286
KingOfCoders wrote: post #3

I have been a paying ChatGPT customer for a long time (since the very beginning). Last week I've compared ChatGPT to Claude and the results (to my eye) were better, the output better structured and the canvas works better. I'm on the edge of jumping ship.

I'd jump ship if it weren't for the real time voice chat. It's extremely powerful for beginner conversation language learning. Hoping that a company will make use of the real time api for a dedicated language learning app soon.

I keep waiting for Anthropic to release real-time voice chat for Claude, too. I don’t know if they’re even working on that, though.

I agree that there need to be dedicated language-learning apps using OpenAI’s realtime API. But at the current pricing—“$0.06 per minute of audio input and $0.24 per minute of audio output” [1]—I don’t think that could be a viable business.

[1] https://openai.com/index/introducing-the-realtime-api/

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#504

Post by bigs »

faizshah wrote: post #501
anotherpaulg wrote: post #416

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

Anecdotally but I still get significantly better results from ChatGPT than claude for coding. Claude is way less controllable it is difficult to get it to do exactly what I want. ChatGPT is way easier to control in terms of asking for specific changes. Not sure why that is maybe the chain of thought and instruction tuning dataset has made theirs a lot better for interactive use.

Try as I might, ChatGPT couldn’t give me working code for a simple admin dash layout in Vue with a sidebar than can minimise. I had to correct it, it would say my apologies and provide new code with a different error. About 10 times in a row it got in a loop of errors and I gave up.

Do any of these actually help coding?

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#505

Post by loktarogar »

lr1970 wrote: post #487

I am curious why "upgraded Claude 3.5 Sonnet" instead of simply Claude 3.6 Sonnet? Minor version increment is a standard way of versioning update. Am i missing something or it is just Anthropic marketing?

Probably because there was no 3.1-3.4, and that the .5 is mostly just to represent that it's an upgrade on Claude 3 but not quite enough to be a Claude 4

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#506

Post by HappMacDonald »

quantadev wrote: post #326
accrual wrote: post #277

Earlier quoted context omitted.

Maybe we could have both - models to improve accessibility (e.g. for users who can't move their body well) and models to perform high level tasks without supervision. It could be very empowering for users with disabilities to regain access computers. But it would also be very powerful to be able to ask "use Photoshop to remove the power lines from this photo" and have the model complete the task and drop off a few sa…

Yep. I agree. The "auto-click" thing would be optional. Should be able to turn it on and off. With auto-click off it would just position the mouse and say "click here".

Cluade scans page and decides which button to click before the screen layout is finished. By the time user authorizes the click, layout has shifted and your click lands on malware advertisements.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#507

Post by famahar »

tkgally wrote: post #503
famahar wrote: post #286

Earlier quoted context omitted.

I'd jump ship if it weren't for the real time voice chat. It's extremely powerful for beginner conversation language learning. Hoping that a company will make use of the real time api for a dedicated language learning app soon.

I keep waiting for Anthropic to release real-time voice chat for Claude, too. I don’t know if they’re even working on that, though. I agree that there need to be dedicated language-learning apps using OpenAI’s realtime API. But at the current pricing—“$0.06 per minute of audio input and $0.24 per minute of audio output” [1]—I don’t think that could be a viable business. [1] https://openai.com/index/introducing-the-re…

Oof yeah that's expensive. Only app I know that will use the API is Speak, which seems to be the most high quality of the AI language learning apps. Seems like they even have a deal with OpenAI. They don't have many languages currently but everyone I know who has used it to learn their target language have improved dramatically.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#509

Post by unshavedyak »

WalterSear wrote: post #411
unshavedyak wrote: post #364

Earlier quoted context omitted.

I feel pain for the people who will be employed to "prompt engineer" the behavior of these things. When they inevitably hallucinate some insane behavior a human will have to take blame for why it's not working.. and yea, that'll be fun to be on the receiving end of.

Humans 'hallucinate' like LLMs. The term used however, is confabulation: we all do it, we all do it quite frequently, and the process is well studied(1). > We are shockingly ignorant of the causes of our own behavior. The explanations that we provide are sometimes wholly fabricated, and certainly never complete. Yet, that is not how it feels. Instead it feels like we know exactly what we're doing and why. This is con…

> Humans 'hallucinate' like LLMs. The term used however, is confabulation: we all do it, we all do it quite frequently, and the process is well studied(1).

Yea i agree, i'm not making a snipe at LLMs or anything of the sort.

I'm saying i expect there to be a human-fallback in the system for quite some time. But solving the fallback problems with be one of black boxes. Which is the worst kind of project in my view, i hate working on code i don't understand. Where the results are not predictable.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#510

Post by runako »

mathnmusic wrote: post #473
runako wrote: post #385

Earlier quoted context omitted.

I asked it to generate a very basic Excel file. It generated text as Markdown. I reiterated that I want an Excel file with formulae and it provided this as part of its response: ---- No, I am not able to generate or create an actual Excel file. As an AI language model, I don't have the capability to create, upload, or send files of any kind, including Excel spreadsheets. ----

It does not have access to the Excel app. You may be able to generate the .xlsx file using python libraries but you would need to run the python code on your own. ChatGPT can run generated code, which is probably why it works there.

I don't expect it to have the Excel app, I expect it to run the code it is capable of generating.

This is what I mean by their strategy being a jumble. Claude can do the hard part of figuring out what code to write and writing it, but then refuses to do the easier part of executing the code.

Post reply on HN