Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Sonnet 4.5

anthropic.com

Post reply on HN
Print view View original

51–60 of 819 posts

Re: Claude Sonnet 4.5

#51

Post by imiric »

_joel wrote: post #42
Kuinox wrote: post #34

Earlier quoted context omitted.

I never understood the point of the pellican on a bicycle exercise: LLMs coding agent doesnt have any way to see the output. It means the only thing this test is testing, is the ability of the LLMs to memorise. Edit: just to show my point, a regular human on a bicycle is way worse with the same model: https://i.imgur.com/flxSJI9.png

Because it excercises thinking about a pelican riding a bike (not common) and then describing that using SVG. It's quite nice imho and seems to scale with the power of the LLM model. Sure Simon has some actual reasons though.

The only thing it exercises is the ability of the model to recall its pelican-on-bicycle and other SVG training data.

Re: Claude Sonnet 4.5

#52

Post by MichealCodes »

I really hope benchmarking improves soon to monitor the model in the weeks following the announcement. It really seems like these companies introduce a new "buffed" model and then slowly nerf the intelligence through optimizations.

If we saw task performance week 1 vs week 8 on benchmarks, this would at least give us more insight into the loop here. In an environment lacking true progress a company could surely "show" it with this strategy.

Re: Claude Sonnet 4.5

#53

Post by ambyra »

yewenjie wrote: post #5

Looking at the chart here, it seems like Sonnet 4 was already better than GPT-5-codex in the SWE verified benchmark. However, my subjective personal experience was GPT-5-codex was far better at complex problems than Claude Code.

For unity gamedev code reviews, I much preferred the gpt5 code. Claude gave me a bunch of bad recommendations for code changes, and also an incorrect formula for completion percentage.

Re: Claude Sonnet 4.5

#54

Post by kruxigt »

AnotherGoodName wrote: post #41
yewenjie wrote: post #5

Looking at the chart here, it seems like Sonnet 4 was already better than GPT-5-codex in the SWE verified benchmark. However, my subjective personal experience was GPT-5-codex was far better at complex problems than Claude Code.

I always wonder how absolute in performance a given model is. Sometimes i ask for Claude-Opus and the responses i get back are worse than the lowest end models of other assistants. Other times it surprises me and is clearly best in class. Sometimes in between this variability of performance it pops up a little survey. "How's Claude doing this session from 1-5? 5 being great." and i suspect i'm in some experiment of e…

[dead]

Re: Claude Sonnet 4.5

#55

Post by Kuinox »

_joel wrote: post #42
Kuinox wrote: post #34

Earlier quoted context omitted.

I never understood the point of the pellican on a bicycle exercise: LLMs coding agent doesnt have any way to see the output. It means the only thing this test is testing, is the ability of the LLMs to memorise. Edit: just to show my point, a regular human on a bicycle is way worse with the same model: https://i.imgur.com/flxSJI9.png

Because it excercises thinking about a pelican riding a bike (not common) and then describing that using SVG. It's quite nice imho and seems to scale with the power of the LLM model. Sure Simon has some actual reasons though.

> Because it excercises thinking about a pelican riding a bike (not common)

It is extremely common, since it's used on every single LLM to bench it.

And there is nothing logic, LLMs are never trained for graphics tasks, they dont see the output of a code.

Re: Claude Sonnet 4.5

#56

Post by blurbleblurble »

yewenjie wrote: post #5

Looking at the chart here, it seems like Sonnet 4 was already better than GPT-5-codex in the SWE verified benchmark. However, my subjective personal experience was GPT-5-codex was far better at complex problems than Claude Code.

GPT-5 is like the guy on the baseball team that's really good at hitting home runs but can't do basic shit in the outfield.

It also consistently gets into drama with the other agents e.g. the other day when I told it we were switching to claude code for executing changes, after badmouthing claude's entirely reasonable and measured analysis it went ahead and decided to `git reset --hard` even after I twice pushed back on that idea.

Whereas gemini and claude are excellent collaborators.

When I do decide to hail mary via GPT-5, I now refer to the other agents as "another agent". But honestly the whole thing has me entirely sketched out.

To be clear, I don't think this was intentionally encoded into GPT-5. What I really think is that OpenAI leadership simply squandered all its good energy and is now coming from behind. Its excellent talent either got demoralized or left.

Re: Claude Sonnet 4.5

#57

Post by zurfer »

zurfer wrote: post #31

Same price and a 4.5 bp jump from 72.7 to 77.2 SWEBench Pretty solid progress for roughly 4 months.

Also getting a perfect score on AIME (math) is pretty cool.

Tongue in cheek: if we progress linearly from here software engineering as defined by SWE bench is solved in 23 months.

Re: Claude Sonnet 4.5

#58

Post by jjcm »

yewenjie wrote: post #5

Looking at the chart here, it seems like Sonnet 4 was already better than GPT-5-codex in the SWE verified benchmark. However, my subjective personal experience was GPT-5-codex was far better at complex problems than Claude Code.

How long have you had early access for?

Re: Claude Sonnet 4.5

#59

Post by adidoit »

cryptoz wrote: post #4

I've really got to refactor my side project which I tailored to just use OpenAI API calls. I think the Anthropic APIs are a bit different so I just never put in the energy to support the changes. I think I remember reading that there are tools to simpify this kind of work, to support multiple LLM APIs? I'm sure I could do it manually but how do you all support multiple API providers that have some differences in the…

LiteLLM is your friend.
Post reply on HN