Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

Post reply on HN
Print view View original

141–150 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#141

Post by root_axis »

cmrdporcupine wrote: post #116
TedDallas wrote: post #99

Earlier quoted context omitted.

Per Anthropic’s RCA linked in Ops post for September 2025 issues: “… To state it plainly: We never reduce model quality due to demand, time of day, or server load. …” So according to Anthropic they are not tweaking quality setting due to demand.

I guess I just don't know how to square that with my actual experiences then. I've seen sporadic drops in reasoning skills that made me feel like it was January 2025, not 2026 ... inconsistent.

I wouldn't doubt that these companies would deliberately degrade performance to manage load, but it's also true that humans are notoriously terrible at identifying random distributions, even with something as simple as a coin flip. It's very possible that what you view as degradation is just "bad RNG".

Re: Claude Code daily benchmarks for degradation tracking

#142

Post by cmrdporcupine »

root_axis wrote: post #141
cmrdporcupine wrote: post #116

Earlier quoted context omitted.

I guess I just don't know how to square that with my actual experiences then. I've seen sporadic drops in reasoning skills that made me feel like it was January 2025, not 2026 ... inconsistent.

I wouldn't doubt that these companies would deliberately degrade performance to manage load, but it's also true that humans are notoriously terrible at identifying random distributions, even with something as simple as a coin flip. It's very possible that what you view as degradation is just "bad RNG".

yep stochastic fantastic

these things are by definition hard to reason about

Re: Claude Code daily benchmarks for degradation tracking

#143

Post by arcanemachiner »

copilot_king wrote: post #96
arcanemachiner wrote: post #71

Earlier quoted context omitted.

Sounds more like a sound business plan than a conspiracy theory.

It sounds like fraud to me

Does it say anywhere in their terms of service that they guarantee the quality of the model, or promise not to modify it?

https://www.anthropic.com/legal/consumer-terms

https://www.anthropic.com/legal/commercial-terms

Re: Claude Code daily benchmarks for degradation tracking

#144

Post by emp17344 »

jerf wrote: post #135
emp17344 wrote: post #123

Earlier quoted context omitted.

Any chance you’re just learning more about what the model is and is not useful for?

I dunno about everyone else but when I learn more about what a model is and is not useful for, my subjective experience improves, not degrades.

Not when the product is marketed as a panacea.

Re: Claude Code daily benchmarks for degradation tracking

#145

Post by AlexandrB »

groundzeros2015 wrote: post #133
rootnod3 wrote: post #105

Earlier quoted context omitted.

And according to Google, they always delete data if requested. And according to Meta, they always give you ALL the data they have on you when requested.

What would you like?

An SLA-style contractually binding agreement.

Re: Claude Code daily benchmarks for degradation tracking

#146

Post by Aurornis »

bgirard wrote: post #120
botacode wrote: post #107

Earlier quoted context omitted.

Load just makes LLMs behave less deterministically and likely degrade. See: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... They don't have to be malicious operators in this case. It just happens.

> malicious It doesn't have to be malicious. If my workflow is to send a prompt once and hopefully accept the result, then degradation matters a lot. If degradation is causing me to silently get worse code output on some of my commits it matters to me. I care about -expected- performance when picking which model to use, not optimal benchmark performance.

Non-determinism isn’t the same as degradation.

The non-determinism means that even with a temperature of 0.0, you can’t expect the outputs to be the same across API calls.

In practice people tend to index to the best results they’ve experienced and view anything else as degradation. In practice it may just be randomness in either direction from the prompts. When you’re getting good results you assume it’s normal. When things feel off you think something abnormal is happening. Rerun the exact same prompts and context with temperature 0 and you might get a different result.

Re: Claude Code daily benchmarks for degradation tracking

#147

Post by [deleted] »

observationist wrote: post #58
sciencejerk wrote: post #11

Why is this happening?

They're "optimizing" costs wherever possible - reducing compute allocations, quantizing models, doing whatever they can to reduce the cost per token, but vehemently insisting that no such things are occurring, that it's all in the users' heads, and using the weaseliest of corporate weasel speak to explain what's happening. They insist it's not happening, then they say something like "oh, it happened but it was an acc…

[deleted]

Re: Claude Code daily benchmarks for degradation tracking

#148

Post by FL33TW00D »

altcognito wrote: post #125
botacode wrote: post #107

Earlier quoted context omitted.

Load just makes LLMs behave less deterministically and likely degrade. See: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... They don't have to be malicious operators in this case. It just happens.

Explain this though. The code is deterministic, even if it relies on pseudo random number generation. It doesn't just happen, someone has to make a conscious decision to force a different code path (or model) if the system is loaded.

It takes a different code path for efficiency.

e.g

if (batch_size > 1024): kernel_x else: kernel_y

Re: Claude Code daily benchmarks for degradation tracking

#149

Post by quadrature »

cmrdporcupine wrote: post #116
TedDallas wrote: post #99

Earlier quoted context omitted.

Per Anthropic’s RCA linked in Ops post for September 2025 issues: “… To state it plainly: We never reduce model quality due to demand, time of day, or server load. …” So according to Anthropic they are not tweaking quality setting due to demand.

I guess I just don't know how to square that with my actual experiences then. I've seen sporadic drops in reasoning skills that made me feel like it was January 2025, not 2026 ... inconsistent.

LLMs sample the next token from a conditional probability distribution, the hope is that dumb sequences are less probable but they will just happen naturally.

Re: Claude Code daily benchmarks for degradation tracking

#150

Post by novaleaf »

bgirard wrote: post #120
botacode wrote: post #107

Earlier quoted context omitted.

Load just makes LLMs behave less deterministically and likely degrade. See: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... They don't have to be malicious operators in this case. It just happens.

> malicious It doesn't have to be malicious. If my workflow is to send a prompt once and hopefully accept the result, then degradation matters a lot. If degradation is causing me to silently get worse code output on some of my commits it matters to me. I care about -expected- performance when picking which model to use, not optimal benchmark performance.

this is about variance of daily statistics, so I think the suggestions are entirely appropriate in this context.
Post reply on HN