Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

271–280 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#271
post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Is there compensation for the tokens because Claude wasted all of them?

So quiet…

Re: Claude Code daily benchmarks for degradation tracking

#272

My personal conspiracy theory is that they choose who to serve a degraded model to based on social graph analysis and sentiment analysis, maximizing for persuasion while minimizing compute.

IMO this strategy seems inspired by TikTok's approach for retaining new uploaders. TikTok used to give new uploaders a visibility boost (i.e., an inflated number of likes and comments) on their first couple of uploads, to get them hooked on the the service. In Anthropic/Claude's case, the strategy is (allegedly) to give new users access to the premium models on sign-up, and then increasingly cut the product with outp…

I would actually assume a little more sophistication. For each user, a measure of "Are they convinced that AI is great". Then, you weaponize your compute to have the maximum social impact. If somebody has a large following (many edges on the social graph), and theyre skeptical of AI tech, inject the expensive but effective models directly into their veins. Let them taste the joy. Then start watering down their dose, and move onto the next person in the graph, again maximizing for net social impact. Language may not even be a consideration

Re: Claude Code daily benchmarks for degradation tracking

#274
If the confidence interval width is 2 * 14.0%, how are you detecting a statistically significant difference between 58% and 50%?

The 95% CIs on both timeseries pretty much always cover the baseline number, which is not consistent with the result being statistically significant.

Re: Claude Code daily benchmarks for degradation tracking

#275
post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Is there compensation for the tokens because Claude wasted all of them?

[deleted]

Re: Claude Code daily benchmarks for degradation tracking

#276

Earlier quoted context omitted.

Explain this though. The code is deterministic, even if it relies on pseudo random number generation. It doesn't just happen, someone has to make a conscious decision to force a different code path (or model) if the system is loaded.

For all practical purposes any code reliant on the output of a PRNG is non-deterministic in all but the most pedantic senses... And if the LLM temperature isn't set to 0 LLMs are sampling from a distribution. If you're going to call a PRNG deterministic then the outcome of a complicated concurrent system with no guaranteed ordering is going to be deterministic too!

Temperature can't be literally zero, or it creates a divide by zero error.

When people say zero, it is shorthand for “as deterministic as this system allows”, but it's still not completely deterministic.

Re: Claude Code daily benchmarks for degradation tracking

#277
post #220
post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Hi. Do you guys have internal degradation tests?

Yes, we do but harnesses are hard to eval, people use them across a huge variety of tasks and sometimes different behaviors tradeoff against each other. We have added some evals to catch this one in particular.

Re: Claude Code daily benchmarks for degradation tracking

#279
post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

[flagged]

Re: Claude Code daily benchmarks for degradation tracking

#280
The degradation does not need to be in the inference it can be in how often inference is used.

It is closed source but the algorithms that decide what Claude code does when, could behave differently when the API responses are slower. Maybe it does fewer investigatory greps or performs fewer tasks to get to “an” answer faster and with less load.

Post reply on HN