Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

anthropic.com

Post reply on HN
Print view View original

441–450 of 758 posts

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#441

Post by aerhardt »

hawk_ wrote: post #433
aerhardt wrote: post #408

Earlier quoted context omitted.

> software errors often cascade Whereas software defects in design and architecture subtly accumulate , until they leave the codebase in a state in which it becomes utterly unworkable. It is one of the chief reasons why good devs get paid what they do. Software discussions very often underrate software extensibility, or in other words, its structural and architectural scaleability. Even software correctness is trivia…

I hear you. But I have wondered if there won't be a need to maintain certain like of software when you can just have it be rewritten for each iteration. Like some kind of schema evolution, yes but throwaway software at each iteration.

Well in terms of processing speed the AI could iterate on different designs until it finds an extensible one, with some kind of reinforcement learning loop. Produce a certain design, get stuck, throw it away, try a new one. Just like humans learn to write good code really - except at an unfathomable speed of iteration. But it still all sounds ridiculously challenging. There is something there that isn't about predicting next tokens like LLMs do. It's about inferring very complex, highly abstract metastructures in the text.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#442

Post by lossolo »

Livebench updated

https://livebench.ai

  Model                         | Global  | Reasoning | Coding  | Math    | Data    | Language | IF

  ------------------------------|---------|-----------|---------|---------|---------|----------|-------

  o1-preview-2024-09-12          | 66.02   | 68.00     | 50.85   | 62.92   | 63.97   | 72.66    | 77.72

  claude-3-5-sonnet-20241022     | 60.33   | 58.67     | 67.13   | 51.28   | 52.78   | 58.09    | 74.05

  claude-3-5-sonnet-20240620     | 59.80   | 58.67     | 60.85   | 53.32   | 56.74   | 56.94    | 72.30

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#443

Post by frankdenbow »

highwaylights wrote: post #81

Completely irrelevant, and it might just be me, but I really like Anthropic's understated branding. OpenAI's branding isn't exactly screaming in your face either, but for something that's generated as much public fear/scaremongering/outrage as LLMs have over the last couple of years, Anthropic's presentation has a much "cosier" veneer to my eyes. This isn't the Skynet Terminator wipe-us-all-out AI, it's the adorable…

This is actually very relevant: most people think this is just an arms race to see who can get the better percentages on benchmarks but to me all this technology is useless if we dont give programmers and end users the right interfaces to utilize the technology.

Anthropic seems to have a better core design and human-computer interaction ethos that shows up all throughout their product and marketing.

I wrote on the topic as well: https://blog.frankdenbow.com/statement-of-purpose/

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#444

Post by cageface »

Bjorkbat wrote: post #345

Tried my standard go-to for testing, asked it to generate a voronoi diagram using p5js. For the sake of job security I'm relieved to see it still can't do a relatively simple task with ample representation in the Google search results. Granted, p5js is kind of niche, but not terribly so. It's arguably the most popular library for creating coding. In case you're wondering, I tried o1-preview, and while it did work, I…

I recently asked it to give me code to do gapless playback of audio files using Apple's AVAudioEngine APIs. It got it wrong and additional prompts to explain why it was wrong didn't help.

To me what it seems like these tools do really well is paraphrase stuff that's in their training data.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#445

Post by zeroonetwothree »

ianeigorndua wrote: post #431
anotherpaulg wrote: post #416

The new Sonnet tops aider's code editing leaderboard at 84.2%. Using aider's "architect" mode it sets the SOTA at 85.7% (with DeepSeek as the "editor" model). 84% Claude 3.5 Sonnet 10/22 80% o1-preview 77% Claude 3.5 Sonnet 06/20 72% DeepSeek V2.5 72% GPT-4o 08/06 71% o1-mini 68% Claude 3 Opus It also sets SOTA on aider's more demanding refactoring benchmark with a score of 92.1%! 92% Sonnet 10/22 75% o1-preview 72%…

Are these synthetic or real-world benchmarks? Answering myself: ”Aider’s code editing benchmark asks the LLM to edit python source files to complete 133 small coding exercises from Exercism” Not gonna start looking for a job any time soon

Example I chose at random:

> Convert a hexadecimal number, represented as a string (e.g. "10af8c"), to its decimal equivalent using first principles (i.e. no, you may not use built-in or external libraries to accomplish the conversion).

So it's fairly synthetic. It's also the sort of thing LLMs should be great at since I'm sure there's tons of data on this sort of thing online.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#446

Post by zeroonetwothree »

nopinsight wrote: post #339

This needs more discussion: Claude using Claude on a computer for coding https://youtu.be/vH2f7cjXjKI?si=Tw7rBPGsavzb-LNo (3 mins) True end-user programming and product manager programming are coming, probably pretty soon. Not the same thing, but Midjourney went from v.1 to v.6 in less than 2 years. If something similar happens, most jobs that could be done remotely will be automatable in a few years.

> If something similar happens, most jobs that could be done remotely will be automatable in a few years.

I'd be willing to be a large amount of money this doesn't happen, assuming "most" means >50% and "a few" is <5.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#447

Post by int_19h »

freediver wrote: post #218

Both new Sonnet and gpt-4o still fail at a simple: "How many w's are in strawberry?" gpt-4o: There are 2 "w's" in "strawberry." Claude 3.5 Sonnet (new): Let me count the w's in "strawberry": 0 w's. (same question with 'r' succeeds) What is artificial about current gen of "artificial intelligence" is the way training (predict next token) and benchmarking (overfitting) is done. Perhaps a fresh approach is needed to ach…

There are models around that will answer this question perfectly.

But why would anyone care? If you want to count w's in a word, that's a Python one-liner, you don't need an LLM for that (although the LLM can write it for you).

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#448

Post by torginus »

modeless wrote: post #111
mmooss wrote: post #87

Of course there's great inefficiency in having the Claude software control a computer with a human GUI mediating everything, but it's necessary for many uses right now given how much we do where only human interfaces are easily accessible. If something like it takes off, I expect interfaces for AI software would be published, standardized, etc. Your customers may not buy software that lacks it. But what I really want…

I hope specialized interfaces for AI never happen. I want AI to use human interfaces, because I want to be empowered to use the same interfaces as AI in the future. A future where only AI can do things because it uses an incomprehensible special interface and the human interface is broken or non-existent is a dystopia. I also want humanoid robots instead of specialized non-humanoid robots for the same reason.

Imo, APIs and to a lesser extent cli tools are already specialized tools made for LLMs.

I've been editing videos with ChatGPT4 + ffmpeg for a year now.

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#449

Post by girvo »

unshavedyak wrote: post #360
nopinsight wrote: post #339

This needs more discussion: Claude using Claude on a computer for coding https://youtu.be/vH2f7cjXjKI?si=Tw7rBPGsavzb-LNo (3 mins) True end-user programming and product manager programming are coming, probably pretty soon. Not the same thing, but Midjourney went from v.1 to v.6 in less than 2 years. If something similar happens, most jobs that could be done remotely will be automatable in a few years.

> If something similar is the case, most jobs that can be done remotely will be automatable in a couple of years. I'm really curious on the cost of that sort of thing. Seems astronomical atm, but as much as i get shocked at the today-cost, staffing is also a pretty insane cost.

Playing with Sonnet 3.5 this morning with Cline, my API cost to add a decent amount of functionality to my GraphQL server cost $0.1325 and took about 5 minutes. $1.80 is a lot cheaper than my hourly rate… but I’m the one reviewing what it doe to ensure it makes sense

And it got some things subtly wrong though so do I/my team. Interesting times ahead I think, but I’m not too worried about my job as a principal dev. Again I’m more stressed about juniors

Re: Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

#450

Post by mnk47 »

throwup238 wrote: post #188
gzer0 wrote: post #182

One of the funnier things during training with the new API (which can control your computer) was this: "Even while recording these demos, we encountered some amusing moments. In one, Claude accidentally stopped a long-running screen recording, causing all footage to be lost. Later, Claude took a break from our coding demo and began to peruse photos of Yellowstone National Park." [0] https://x.com/AnthropicAI/status/1…

At least now we know SkyClaude’s plan to end human civilization. It’s planning on triggering a Yellowstone caldera super eruption.

Am I misremembering or is this an exact plot point of Pluto (the manga/anime)?
Post reply on HN