Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.
Is there compensation for the tokens because Claude wasted all of them?
Claude Code daily benchmarks for degradation tracking
271–280 of 372 posts
Re: Claude Code daily benchmarks for degradation tracking
#272My personal conspiracy theory is that they choose who to serve a degraded model to based on social graph analysis and sentiment analysis, maximizing for persuasion while minimizing compute.
IMO this strategy seems inspired by TikTok's approach for retaining new uploaders. TikTok used to give new uploaders a visibility boost (i.e., an inflated number of likes and comments) on their first couple of uploads, to get them hooked on the the service. In Anthropic/Claude's case, the strategy is (allegedly) to give new users access to the premium models on sign-up, and then increasingly cut the product with outp…
Re: Claude Code daily benchmarks for degradation tracking
#273Re: Claude Code daily benchmarks for degradation tracking
#274The 95% CIs on both timeseries pretty much always cover the baseline number, which is not consistent with the result being statistically significant.
Re: Claude Code daily benchmarks for degradation tracking
#275Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.
Is there compensation for the tokens because Claude wasted all of them?
Re: Claude Code daily benchmarks for degradation tracking
#276Earlier quoted context omitted.
Explain this though. The code is deterministic, even if it relies on pseudo random number generation. It doesn't just happen, someone has to make a conscious decision to force a different code path (or model) if the system is loaded.
For all practical purposes any code reliant on the output of a PRNG is non-deterministic in all but the most pedantic senses... And if the LLM temperature isn't set to 0 LLMs are sampling from a distribution. If you're going to call a PRNG deterministic then the outcome of a complicated concurrent system with no guaranteed ordering is going to be deterministic too!
When people say zero, it is shorthand for “as deterministic as this system allows”, but it's still not completely deterministic.
Re: Claude Code daily benchmarks for degradation tracking
#277Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.
Hi. Do you guys have internal degradation tests?
Re: Claude Code daily benchmarks for degradation tracking
#278Re: Claude Code daily benchmarks for degradation tracking
#279Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.
Re: Claude Code daily benchmarks for degradation tracking
#280It is closed source but the algorithms that decide what Claude code does when, could behave differently when the API responses are slower. Maybe it does fewer investigatory greps or performs fewer tasks to get to “an” answer faster and with less load.