Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

Post reply on HN
Print view View original

101–110 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#101

Post by rootnod3 »

ofirpress wrote: post #24

[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.

Sorry what?

"You can't measure my Cloud Service's performance correctly if my servers are overloaded"?

"Oh, you just measured me at bad times each day. On only 50 different queries."

So, what does that mean? I have to pick specific times during the day for Claude to code better?

Does Claude Code have office hours basically?

Re: Claude Code daily benchmarks for degradation tracking

#103

Post by copilot_king »

rootnod3 wrote: post #101
ofirpress wrote: post #24

[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.

Sorry what? "You can't measure my Cloud Service's performance correctly if my servers are overloaded"? "Oh, you just measured me at bad times each day. On only 50 different queries." So, what does that mean? I have to pick specific times during the day for Claude to code better? Does Claude Code have office hours basically?

[flagged]

Re: Claude Code daily benchmarks for degradation tracking

#104

Post by warkdarrior »

levkk wrote: post #100
antirez wrote: post #29

Why I do not believe this shows Anthropic serves folks a worse model: 1. The percentage drop is too low and oscillating, it goes up and down. 2. The baseline of Sonnet 4.5 (the obvious choice for when they have GPU busy for the next training) should be established to see Opus at some point goes Sonnet level. This was not done but likely we would see a much sharp decline in certain days / periods. The graph would look…

I believe the science, but I've been using it daily and it's been getting worse, noticeably.

Is it possible that your expectations are increasing, not that the model is getting worse?

Re: Claude Code daily benchmarks for degradation tracking

#105

Post by rootnod3 »

TedDallas wrote: post #99
codeflo wrote: post #79

Earlier quoted context omitted.

This is intentional? I think delivering lower quality than what was advertised and benchmarked is borderline fraud, but YMMV.

Per Anthropic’s RCA linked in Ops post for September 2025 issues: “… To state it plainly: We never reduce model quality due to demand, time of day, or server load. …” So according to Anthropic they are not tweaking quality setting due to demand.

And according to Google, they always delete data if requested.

And according to Meta, they always give you ALL the data they have on you when requested.

Re: Claude Code daily benchmarks for degradation tracking

#106

Post by rootnod3 »

copilot_king wrote: post #94
codeflo wrote: post #79

Earlier quoted context omitted.

This is intentional? I think delivering lower quality than what was advertised and benchmarked is borderline fraud, but YMMV.

[flagged]

That number is a sliding window, isn't it?

Re: Claude Code daily benchmarks for degradation tracking

#107

Post by botacode »

Davidzheng wrote: post #43
ofirpress wrote: post #24

[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.

but degradation from servers being overloaded would be the type of degradation this SHOULD measure no? Unless it's only intended for measuring their quietly distilling models (which they claim not to do? idk for certain)

Load just makes LLMs behave less deterministically and likely degrade. See: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

They don't have to be malicious operators in this case. It just happens.

Re: Claude Code daily benchmarks for degradation tracking

#108

Post by direwolf20 »

codeflo wrote: post #79
exitb wrote: post #66

Earlier quoted context omitted.

An operator at load capacity can either refuse requests, or move the knobs (quantization, thinking time) so requests process faster. Both of those things make customers unhappy, but only one is obvious.

This is intentional? I think delivering lower quality than what was advertised and benchmarked is borderline fraud, but YMMV.

They don't advertise a certain quality. You take what they have or leave it.

Re: Claude Code daily benchmarks for degradation tracking

#109

Post by rootnod3 »

copilot_king wrote: post #103
rootnod3 wrote: post #101

Earlier quoted context omitted.

Sorry what? "You can't measure my Cloud Service's performance correctly if my servers are overloaded"? "Oh, you just measured me at bad times each day. On only 50 different queries." So, what does that mean? I have to pick specific times during the day for Claude to code better? Does Claude Code have office hours basically?

[flagged]

Verily, my vichyssoise of verbiage veers most verbose, so let me run that thing out of tokens fast.

Re: Claude Code daily benchmarks for degradation tracking

#110

Post by alias_neo »

observationist wrote: post #62
giwook wrote: post #16

Earlier quoted context omitted.

https://www.anthropic.com/engineering/a-postmortem-of-three-...

>>> We never reduce model quality due to demand, time of day, or server load. The problems our users reported were due to infrastructure bugs alone. Just ignore the continual degradation of service day over day, long after the "infrastructure bugs" have reportedly been solved. Oh, and I've got a bridge in Brooklyn to sell ya, it's a great deal!

> We never reduce model quality due to demand, time of day, or server load

Forgive me, but as a native English speaker, this sentence says exactly one thing to me; We _do_ reduce model quality, just not for these listed reasons.

If they don't do it, they could put a full stop after the fifth word and save some ~~tokens~~ time.

Post reply on HN