Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Opus 4.5

anthropic.com

Post reply on HN
Print view View original

241–250 of 525 posts

Re: Claude Opus 4.5

#241

Post by zsoltkacsandi »

lukev wrote: post #227
zsoltkacsandi wrote: post #197

Earlier quoted context omitted.

> The nerf is psychologial, not actual Once I tested this, I gave the same task for a model after the release and a couple weeks later. In the first attempt it produced a well-written code that worked beautifully, I started to worry about the jobs of the software engineers. Second attempt was a nightmare, like a butcher acting as a junior developer performing a surgery on a horse. Is this empirical evidence? And this…

> Is this empirical evidence? Look, I'm not defending the big labs, I think they're terrible in a lot of ways. And I'm actually suspending judgement on whether there is ~some kind of nerf happening. But the anecdote you're describing is the definition of non-empirical. It is entirely subjective, based entirely on your experience and personal assessment.

> But the anecdote you're describing is the definition of non-empirical. It is entirely subjective, based entirely on your experience and personal assessment.

Well, if we see this way, this is true for Antrophic’s benchmarks as well.

Btw the definition of empirical is: “based on observation or experience rather than theory or pure logic”

So what I described is the exact definition of empirical.

Re: Claude Opus 4.5

#242

Post by davidsainez »

lukev wrote: post #154
unsupp0rted wrote: post #46

This is gonna be game-changing for the next 2-4 weeks before they nerf the model. Then for the next 2-3 months people complaining about the degradation will be labeled “skill issue”. Then a sacrificial Anthropic engineer will “discover” a couple obscure bugs that “in some cases” might have lead to less than optimal performance. Still largely a user skill issue though. Then a couple months later they’ll release Opus 4…

There are two possible explanations for this behavior: the model nerf is real, or there's a perceptual/psychological shift. However, benchmarks exist. And I haven't seen any empirical evidence that the performance of a given model version grows worse over time on benchmarks (in general.) Therefore, some combination of two things are true: 1. The nerf is psychologial, not actual. 2. The nerf is real but in a way that…

There are well documented cases of performance degradation: https://www.anthropic.com/engineering/a-postmortem-of-three-....

The real issue is that there is no reliable system currently in place for the end user (other than being willing to burn the cash and run your own benchmarks regularly) to detect changes in performance.

It feels to me like a perfect storm. A combination of high cost of inference, extreme competition, and the statistical nature of LLMs make it very tempting for a provider to tune their infrastructure in order to squeeze more volume from their hardware. I don't mean to imply bad faith actors: things are moving at breakneck speed and people are trying anything that sticks. But the problem persists, people are building on systems that are in constant flux (for better or for worse).

Re: Claude Opus 4.5

#245

Post by maherbeg »

Ok, the victorian lock puzzle game is pretty damn cool way to showcase the capabilities of these models. I kinda want to start building similar puzzle games for models to solve.

Re: Claude Opus 4.5

#246

Post by Wowfunhappy »

davidsainez wrote: post #242
lukev wrote: post #154

Earlier quoted context omitted.

There are two possible explanations for this behavior: the model nerf is real, or there's a perceptual/psychological shift. However, benchmarks exist. And I haven't seen any empirical evidence that the performance of a given model version grows worse over time on benchmarks (in general.) Therefore, some combination of two things are true: 1. The nerf is psychologial, not actual. 2. The nerf is real but in a way that…

There are well documented cases of performance degradation: https://www.anthropic.com/engineering/a-postmortem-of-three-... . The real issue is that there is no reliable system currently in place for the end user (other than being willing to burn the cash and run your own benchmarks regularly) to detect changes in performance. It feels to me like a perfect storm. A combination of high cost of inference, extreme compe…

> There are well documented cases of performance degradation: https://www.anthropic.com/engineering/a-postmortem-of-three-...

There was one well-documented case of performance degradation which arose from a stupid bug, not some secret cost cutting measure.

Re: Claude Opus 4.5

#247

Post by morgengold »

I'm on a Claude Code Max subscription. Last days have been a struggle with Sonnet 4.5 - Now it switched to Claude Opus 4.5 as default model. Ridiculous good and fast.

Re: Claude Opus 4.5

#248

Post by baq »

zsoltkacsandi wrote: post #236
pertymcpert wrote: post #229

Earlier quoted context omitted.

The models are non-deterministic. You can't just assume that because it did better before that it was on average better than before. And the variance is quite large.

No one talked about determinism. First it was able to do a task, second time not. It’s not that the implementation details changed.

This isn’t how you should be benchmarking models. You should give it the same task n times and see how often it succeeds and/or how long it takes to be successful (see also the 50% time horizon metric by METR).

Re: Claude Opus 4.5

#249

Post by dave1010uk »

The Claude Opus 4.5 system card [0] is much more revealing than the marketing blog post. It's a 150 page PDF, with all sorts of info, not just the usual benchmarks.

There's a big section on deception. One example is Opus is fed news about Anthropic's safety team being disbanded but then hides that info from the user.

The risks are a bit scary, especially around CBRNs. Opus is still only ASL-3 (systems that substantially increase the risk of catastrophic misuse) and not quite at ASL-4 (uplifting a second-tier state-level bioweapons programme to the sophistication and success of a first-tier one), so I think we're fine...

I've never written a blog post about a model release before but decided to this time [1]. The system card has quite a few surprises, so I've highlighted some bits that stood out to me (and Claude, ChatGPT and Gemini).

[0] https://www.anthropic.com/claude-opus-4-5-system-card

[1] https://dave.engineer/blog/2025/11/claude-opus-4.5-system-ca...

Re: Claude Opus 4.5

#250

Post by zwnow »

llamasushi wrote: post #30

The burying of the lede here is insane. $5/$25 per MTok is a 3x price drop from Opus 4. At that price point, Opus stops being "the model you use for important things" and becomes actually viable for production workloads. Also notable: they're claiming SOTA prompt injection resistance. The industry has largely given up on solving this problem through training alone, so if the numbers in the system card hold up under a…

Why do all these comments sound like a sales pitch? Everytime some new bullshit model is released there are hundreds of comments like this one, pointing out 2 features talking about how huge all of this is. It isn't.
Post reply on HN