Tried Kimi 2.5 and far ahead of claude for coding.
Claude Code daily benchmarks for degradation tracking
331–340 of 372 posts
Re: Claude Code daily benchmarks for degradation tracking
#332Earlier quoted context omitted.
I believe the science, but I've been using it daily and it's been getting worse, noticeably.
I’m finding Gemini and chatGPT web terminal to out perform Claude code. The context becomes too much for the LLM, and tries to make up for it by doing more file read ops.
Sounds like you might want to refactor the code if the individual files are too big and it can't find what it's looking for?
Re: Claude Code daily benchmarks for degradation tracking
#333Why I do not believe this shows Anthropic serves folks a worse model: 1. The percentage drop is too low and oscillating, it goes up and down. 2. The baseline of Sonnet 4.5 (the obvious choice for when they have GPU busy for the next training) should be established to see Opus at some point goes Sonnet level. This was not done but likely we would see a much sharp decline in certain days / periods. The graph would look…
I too suspect the A/B testing is the prime suspect: context window limits, system prompts, MAYBE some other questionable things that should be disclosed. Either way, if true, given the cost I wish I could opt-out or it were more transparent. Put out variants you can select and see which one people flock to. I and many others would probably test constantly and provide detailed feedback. All speculation though
If that's the case, then as a benchmark operator you'd want to run the benchmark through multiple different accounts on different machines to average over A/B test noise.
Re: Claude Code daily benchmarks for degradation tracking
#334Earlier quoted context omitted.
One thing that could be a strong degradation especially for benchmarks is they switched the default "Exit Plan" mode from: "Proceed" to "Clear Context and Proceed" It's rare you'd want to do that unless you're actually near the context window after planning. I pressed it accidentally once, and it managed to forget one of the clarifying questions it asked me because it hadn't properly written that to the plan file. If…
I disagree that this was the issue, or that it's "rare that you'd want to do that unless you're near the context window". Clearing context after writing a plan, before starting implementation of said plan, is common practice (probably standard practice) with spec driven development. If the plan is adequate, then compaction would be redundant.
For a 2M+ LOC codebase, the plans alone are never adequate. They miss nuance that the agent will only have to rediscover when it comes to operate on them.
For spec driven development (which I do for larger issues), this badly affects the plan to generate the spec, not the spec itself.
I'll typically put it in plan mode, and ask it to generate documentation about an issue or feature request.
When it comes to write the output to the .typ file, it does much much worse if it has a cleared context and a plan file than if it has it's full context.
The previously "thought" is typically, "I know what to write now, let me exit plan mode".
Clearing context on exiting that plan mode is a disaster which leaves you much worse off and skeletal documentation and specs compared to letting it flow.
A new context to then actually implement the documented spec is not so bad, although I'd still rather compact.
Re: Claude Code daily benchmarks for degradation tracking
#335Earlier quoted context omitted.
Load just makes LLMs behave less deterministically and likely degrade. See: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... They don't have to be malicious operators in this case. It just happens.
The primary (non malicious, non stupid) explanation given here is batching. But I think you would find looking at large-scale inference the batch sizes being ran on any given rig are fairly static - there is a sweet spot for any given model part ran individually between memory consumption and GPU utilization, and generally GPUs do badly at job parallelism. I think the more likely explanation is again with the extreme…
Why do you think batching has anything to do with the model getting dumber? Do you know what batching means?
Re: Claude Code daily benchmarks for degradation tracking
#336[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.
Why should users care about Anthropic's servers being overloaded?
Re: Claude Code daily benchmarks for degradation tracking
#337Earlier quoted context omitted.
Interesting. On first glance that seems over engineered. I wonder what the reason is for doing it that way?
If you don't do it that way then resizing the terminal corrupts what's on screen.
It's possible to handle resizes without all this machinery, most simply by clearing the screen and redrawing everything when a resize occurs. Some TUI libraries will automatically do this for you.
Programs like top, emacs, tmux, etc are most definitely not implemented using this stack, yet they handle resizing just fine.
Re: Claude Code daily benchmarks for degradation tracking
#338Earlier quoted context omitted.
Is there compensation for the tokens because Claude wasted all of them?
You’re lucky they have even admitted a problem instead of remaining silent and quietly fixing it. Do not expect ethical behaviour from this company.
Why not, can you expand? Asking because I’m considering Claude due to the sandbox feature.
Re: Claude Code daily benchmarks for degradation tracking
#339Earlier quoted context omitted.
You joke but having CC open in the terminal hits 10% on my gpu to render the spinning thinking animation for some reason. Switch out of the terminal tab and gpu drops back to zero.
That sounds like an issue with your terminal more than an issue with CC...
[dead]
Re: Claude Code daily benchmarks for degradation tracking
#340Earlier quoted context omitted.
Hi. Do you guys have internal degradation tests?
Yes, we do but harnesses are hard to eval, people use them across a huge variety of tasks and sometimes different behaviors tradeoff against each other. We have added some evals to catch this one in particular.
Can't you keep the model the same, until the user chooses to use a different model?