Wouldn't be surprised if they slowly start quantizing their models over time. Makes it easier to scale and reduce operational cost. Also makes a new release have more impact as it will be more notably "better" than what you've been using the past couple of days/weeks.
Claude Code daily benchmarks for degradation tracking
21–30 of 372 posts
Re: Claude Code daily benchmarks for degradation tracking
#22Simply search user prompts for curse words and then measure hostility sentiment. User hostility rises as agents fail to meet expectations.
I feel bad about it but sometimes it's so daft, I can't even xD
It's not my fault, they set high standards!
Re: Claude Code daily benchmarks for degradation tracking
#23Would love to see this idea expanded to ever alleged SoTA model currently in production. Any speculation as to why this degradation occurs?
Anecdote, I don't have any proof and it's just a feeling. But around afternoon in GMT+1 compared to the morning/midday, there seems to be a change in the quality of responses, which seems to line up with when the US wakes up. I consistently get (what feels like) worse responses in both Codex and Claude Code in the afternoon/night compared to morning/midday, so much that I usually give up then try the same prompt next…
It’s the afternoon slump. The AI needs a cup of coffee and to doomscroll for half an hour!
Re: Claude Code daily benchmarks for degradation tracking
#24[SWE-bench co-author here]
It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that.
I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.
Re: Claude Code daily benchmarks for degradation tracking
#25Earlier quoted context omitted.
Anecdote, I don't have any proof and it's just a feeling. But around afternoon in GMT+1 compared to the morning/midday, there seems to be a change in the quality of responses, which seems to line up with when the US wakes up. I consistently get (what feels like) worse responses in both Codex and Claude Code in the afternoon/night compared to morning/midday, so much that I usually give up then try the same prompt next…
It’s the afternoon slump. The AI needs a cup of coffee and to doomscroll for half an hour!
Or a load balancing technique :) Either way, it kicks me off to do other things so maybe it isn't so bad after all.
Re: Claude Code daily benchmarks for degradation tracking
#26[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.
Hope you don't mind the unrelated question:
How do you pay for those SWE-bench runs?
I am trying to run a benchmark but it is too expensive to run enough runs to get a fair comparison.
Re: Claude Code daily benchmarks for degradation tracking
#27In medicine there is a concept of reporting adverse effects of medication or interventions which are then collectively studied for Public Health [MedWatch][VAERS][EudraVigilance] and in academia. We should have something like that for all coding agents(and agents in other fields too), given how widely its deployed and affect on "health" in general(not only human). Call it the AI "health" of things benchmark.
I would imagine a sort of hybrid qualities of volunteer efforts like wikipedia, new problems like advent of code and benchmarks like this. The goal? It would be to study the collective effort on the affects of usage to so many areas where AI is used.
[MedWatch](https://www.fda.gov/safety/medwatch-fda-safety-information-a...)
[VAERS](https://www.cdc.gov/vaccine-safety-systems/vaers/index.html)
[EudraVigilance](https://www.ema.europa.eu/en/human-regulatory-overview/resea...)
Re: Claude Code daily benchmarks for degradation tracking
#28Earlier quoted context omitted.
That's why benchmarks are useful. We all suffer from the shortcomings of human perception.
Benchmarks shortcomings are no worse... they inevitably measure something that is only close to the thing you actually care about, not the thing you actually care about. It's entirely plausible that this decreased benchmark score is because Anthropic's initial prompting of the model was overtuned to the benchmark and as they're gaining more experience with real world use they are changing the prompt to do better at t…
I wonder how best we can measure the usefulness of models going forward.
Thumbs up or down? (could be useful for trends) Usage growth from the same user over time? (as an approximation) Tone of user responses? (Don't do this... this is the wrong path... etc.)
Re: Claude Code daily benchmarks for degradation tracking
#29Why I do not believe this shows Anthropic serves folks a worse model:
1. The percentage drop is too low and oscillating, it goes up and down.
2. The baseline of Sonnet 4.5 (the obvious choice for when they have GPU busy for the next training) should be established to see Opus at some point goes Sonnet level. This was not done but likely we would see a much sharp decline in certain days / periods. The graph would look like dominated by a "square wave" shape.
3. There are much better explanations for this oscillation: A) They have multiple checkpoints and are A/B testing, CC asks you feedbacks about the session. B) Claude Code itself gets updated, as the exact tools version the agent can use change. In part it is the natural variability due to the token sampling that makes runs not equivalent (sometimes it makes suboptimal decisions compared to T=0) other than not deterministic, but this is the price to pay to have some variability.
Re: Claude Code daily benchmarks for degradation tracking
#30Simply search user prompts for curse words and then measure hostility sentiment. User hostility rises as agents fail to meet expectations.
there are many times where I just do it myself and it thinks it did well.