Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

Post reply on HN
Print view View original

201–210 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#201

Post by dingnuts »

Aurornis wrote: post #146
bgirard wrote: post #120

Earlier quoted context omitted.

> malicious It doesn't have to be malicious. If my workflow is to send a prompt once and hopefully accept the result, then degradation matters a lot. If degradation is causing me to silently get worse code output on some of my commits it matters to me. I care about -expected- performance when picking which model to use, not optimal benchmark performance.

Non-determinism isn’t the same as degradation. The non-determinism means that even with a temperature of 0.0, you can’t expect the outputs to be the same across API calls. In practice people tend to index to the best results they’ve experienced and view anything else as degradation. In practice it may just be randomness in either direction from the prompts. When you’re getting good results you assume it’s normal. Whe…

[dead]

Re: Claude Code daily benchmarks for degradation tracking

#202

Post by embedding-shape »

chrisjj wrote: post #93
Wheaties466 wrote: post #63

Earlier quoted context omitted.

from what I understand this can come from the batching of requests.

So, a known bug?

No, basically, the requests are processed in batches, together, and the order they're listed in matters for the results, as the grid (tiles) that the GPU is ultimately processing, are different depending on what order they entered at.

So if you want batching + determinism, you need the same batch with the same order which obviously don't work when there are N+1 clients instead of just one.

Re: Claude Code daily benchmarks for degradation tracking

#203

Post by vidarh »

chrisjj wrote: post #92
vidarh wrote: post #60

Earlier quoted context omitted.

It would happen if they quietly decide to serve up more aggressively distilled / quantised / smaller models when under load.

They advertise the Opus 4.5 model. Secretly substituting a cheaper one to save costs would be fraud.

If you use the API, you pay for a specific model, yes, but even then there are "workarounds" for them, such as someone else pointed out by reducing the amount of time they let it "think".

If you use the subscriptions, the terms specifically says that beyond the caps they can limit your "model and feature usage, at our discretion".

Re: Claude Code daily benchmarks for degradation tracking

#204

Post by persedes »

What would be cool if this somehow could do a comparison by provider. E.g. in the last outages anthropic models running on vertex were apparently less affected than those deployed elsewhere. (Not saying that one is better than the other, but would be a neat read out).

Re: Claude Code daily benchmarks for degradation tracking

#205

Post by chrisjj »

embedding-shape wrote: post #202
chrisjj wrote: post #93

Earlier quoted context omitted.

So, a known bug?

No, basically, the requests are processed in batches, together, and the order they're listed in matters for the results, as the grid (tiles) that the GPU is ultimately processing, are different depending on what order they entered at. So if you want batching + determinism, you need the same batch with the same order which obviously don't work when there are N+1 clients instead of just one.

Sure, but how can that lead to increased demand resulting in decreased intelligence? That is the effect we are discussing.

Re: Claude Code daily benchmarks for degradation tracking

#206

Post by isaacdl »

trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Anywhere we can read more about what a "harness issue" means? What was the impact of it?

Re: Claude Code daily benchmarks for degradation tracking

#207

Post by hatmanstack »

stefan_ wrote: post #173
botacode wrote: post #107

Earlier quoted context omitted.

Load just makes LLMs behave less deterministically and likely degrade. See: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... They don't have to be malicious operators in this case. It just happens.

The primary (non malicious, non stupid) explanation given here is batching. But I think you would find looking at large-scale inference the batch sizes being ran on any given rig are fairly static - there is a sweet spot for any given model part ran individually between memory consumption and GPU utilization, and generally GPUs do badly at job parallelism. I think the more likely explanation is again with the extreme…

That's why I'd love to get stats on load/hardware/location of where my inference is running. Looking at you Trainiuim.

Re: Claude Code daily benchmarks for degradation tracking

#208

Post by chrisjj »

vidarh wrote: post #203
chrisjj wrote: post #92

Earlier quoted context omitted.

They advertise the Opus 4.5 model. Secretly substituting a cheaper one to save costs would be fraud.

If you use the API, you pay for a specific model, yes, but even then there are "workarounds" for them, such as someone else pointed out by reducing the amount of time they let it "think". If you use the subscriptions, the terms specifically says that beyond the caps they can limit your "model and feature usage, at our discretion".

Sure. I was separating the model - which Anthropic promises not to downgrade - and the "thinking time" - which Anthropic doesn't promise not to downgrade. It seems the latter is very likely the culprit in this case.

Re: Claude Code daily benchmarks for degradation tracking

#209

Post by conception »

davidee wrote: post #169
levkk wrote: post #100

Earlier quoted context omitted.

I believe the science, but I've been using it daily and it's been getting worse, noticeably.

I have to concur. And to the question about understanding what its good and bad at; no, tasks that it could accomplish quickly and easily just a month ago, now require more detailed prompting and constant "erroneous direction correction." It's almost as if, as tool use and planning capabilities have expanded, Claude (as a singular product) is having a harder time coming up with simple approaches that just work, inste…

I assume, after any compacting of the context window that the session is more or less useless at that point I’ve never had consistent results after compacting.

Re: Claude Code daily benchmarks for degradation tracking

#210

Post by jmalicki »

altcognito wrote: post #125
botacode wrote: post #107

Earlier quoted context omitted.

Load just makes LLMs behave less deterministically and likely degrade. See: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... They don't have to be malicious operators in this case. It just happens.

Explain this though. The code is deterministic, even if it relies on pseudo random number generation. It doesn't just happen, someone has to make a conscious decision to force a different code path (or model) if the system is loaded.

For all practical purposes any code reliant on the output of a PRNG is non-deterministic in all but the most pedantic senses... And if the LLM temperature isn't set to 0 LLMs are sampling from a distribution.

If you're going to call a PRNG deterministic then the outcome of a complicated concurrent system with no guaranteed ordering is going to be deterministic too!

Post reply on HN