Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

Post reply on HN
Print view View original

221–230 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#221

Post by plagiarist »

jabedude wrote: post #164
plagiarist wrote: post #117

Earlier quoted context omitted.

IMO it should need a third party running the LLM anyway. Otherwise the evaluated company could notice they're receiving the same requests daily and discover benchmarking that way.

But that's removing a component that's critical for the test. We as users/benchmark consumers care that the service as provided by Anthropic/OpenAI/Google is consistent over time given the same model/prompt/context

Might as well have the free tokens, then, especially if it is an open benchmark they are already aware of. If they want to game it they cannot be stopped from doing so when it's on their infra.

Re: Claude Code daily benchmarks for degradation tracking

#222

Post by embedding-shape »

chrisjj wrote: post #218
embedding-shape wrote: post #214

Earlier quoted context omitted.

Small subtle errors that are only exposed at certain execution parts could be one. You might place things differently onto the GPU depending on how large the batch is, if you've found one way to be faster batch_size 1024. As number of concurrent incoming requests goes up, you increase batch_size. Just one possibility, guess there could be a multitude of reasons, as it's really hard to reason about until you sit with…

Wouldn't you think that was as likely to increase as decrease intelligence, so average to nil in the benchmarks?

No, I'm not sure how that'd make sense. Either you're making the correct (expected) calculations, or you're getting it wrong. Depending the type of wrong or how wrong, could go from "used #2 in attention instead of #1" so "blue" instead of "Blue" or whatever, to completely incoherent text and garbled output.

Re: Claude Code daily benchmarks for degradation tracking

#223

Post by [deleted] »

arcanemachiner wrote: post #143
copilot_king wrote: post #96

Earlier quoted context omitted.

It sounds like fraud to me

Does it say anywhere in their terms of service that they guarantee the quality of the model, or promise not to modify it? https://www.anthropic.com/legal/consumer-terms https://www.anthropic.com/legal/commercial-terms

[deleted]

Re: Claude Code daily benchmarks for degradation tracking

#224

Post by awestroke »

hu3 wrote: post #220
trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Hi. Do you guys have internal degradation tests?

[flagged]

Re: Claude Code daily benchmarks for degradation tracking

#225

Post by measurablefunc »

kevin_thibedeau wrote: post #184
measurablefunc wrote: post #165

Earlier quoted context omitted.

That would just add up to statistical noise instead of 10% degradation over a week.

Catastrophic error accumulation can produce more profound effects than noise.

Just to make sure I got this right. They serve millions of requests a day & somehow catastrophic error accumulation is what is causing the 10% degradation & no one at Anthropic is noticing it. Is that the theory?

Re: Claude Code daily benchmarks for degradation tracking

#228

Post by chrisjj »

embedding-shape wrote: post #222
chrisjj wrote: post #218

Earlier quoted context omitted.

Wouldn't you think that was as likely to increase as decrease intelligence, so average to nil in the benchmarks?

No, I'm not sure how that'd make sense. Either you're making the correct (expected) calculations, or you're getting it wrong. Depending the type of wrong or how wrong, could go from "used #2 in attention instead of #1" so "blue" instead of "Blue" or whatever, to completely incoherent text and garbled output.

I accept errors are more likely to decrease "intelligence". But I don't see how increased load, through batching, is any more likely to increase than decrease errors.

Re: Claude Code daily benchmarks for degradation tracking

#230

Post by kristianp »

dajonker wrote: post #21

Wouldn't be surprised if they slowly start quantizing their models over time. Makes it easier to scale and reduce operational cost. Also makes a new release have more impact as it will be more notably "better" than what you've been using the past couple of days/weeks.

Open weights models such as GPT-OSS, Kimi K2.x are trained with 4 bit layers. So it wouldn't come as a surprise if the closed models do similar things. If I compare Kimi K2.5 and Opus 4.5 on openrouter, output tokens are about 8x more expensive for Opus, which might indicate Opus is much larger and doesn't quantize, but the claude subscription plans muddy the waters on price comparison a lot.
Post reply on HN