Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

Post reply on HN
Print view View original

291–300 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#291

Post by TZubiri »

stavros wrote: post #282
easygenes wrote: post #264

Earlier quoted context omitted.

He doesn't: https://x.com/trq212/status/2014051501786931427

What? Technology has stopped making sense to me. Drawing a UI with React and rasterizing it to ANSI? Are we competing to see what the least appropriate use of React is? Are they really using React to draw a few boxes of text on screen? I'm just flabbergasted.

It's AI all the way down

But it's very subsidizes when compared to API tokens, so we are all being paid by VCs to write prompts actually.

Re: Claude Code daily benchmarks for degradation tracking

#292

Post by gizmodo59 »

samlinnfer wrote: post #269
trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Is there compensation for the tokens because Claude wasted all of them?

Codex seems to give compensation tokens whenever this happens! Hope Claude gives too.

Re: Claude Code daily benchmarks for degradation tracking

#293

Post by jusgu »

macinjosh wrote: post #279
trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

[flagged]

the issue is unrelated to the foundational model but rather the prompts and tool calling that encapsulate the model

Re: Claude Code daily benchmarks for degradation tracking

#294

Post by mathrawka »

samlinnfer wrote: post #269
trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Is there compensation for the tokens because Claude wasted all of them?

You are funny. Anthropic refuses to issue refunds, even when they break things.

I had an API token set via an env var on my shell, and claude code changed to read that env var. I had a $10 limit set on it, so found out it was using the API, instead of my subscription, when it stopped working.

I filed a ticket and they refused to refund me, even though it was a breaking change with claude code.

Re: Claude Code daily benchmarks for degradation tracking

#295

Post by forgotTheLast »

joquarky wrote: post #276
jmalicki wrote: post #210

Earlier quoted context omitted.

For all practical purposes any code reliant on the output of a PRNG is non-deterministic in all but the most pedantic senses... And if the LLM temperature isn't set to 0 LLMs are sampling from a distribution. If you're going to call a PRNG deterministic then the outcome of a complicated concurrent system with no guaranteed ordering is going to be deterministic too!

Temperature can't be literally zero, or it creates a divide by zero error. When people say zero, it is shorthand for “as deterministic as this system allows”, but it's still not completely deterministic.

Zero temp just uses argmax, which is what softmax approaches if you take the limit of T to zero anyway. So it could very well be deterministic.

Re: Claude Code daily benchmarks for degradation tracking

#296

Post by abathologist »

cmrdporcupine wrote: post #45
yoavsha1 wrote: post #34

Earlier quoted context omitted.

I had that exact same feeling during the US holidays where I got to enjoy 2x usage limits and everything just seemed to work well

I had terrible results during the holidays -- it wasn't slow but it was clear they were dealing with the load by quantizing in spots because there were entire chunks of days when the results from it were so terrible I gave up and switched to using Gemini or Codex via opencode.

I find that if I have my rabbit's foot and lucky socks on, I win working code ~1.2x more often.

Re: Claude Code daily benchmarks for degradation tracking

#297

Post by bushbaba »

hu3 wrote: post #220
trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Hi. Do you guys have internal degradation tests?

I’d wager probably not. It’s not like reliability is what will get them marketshare. And the fast pace of industry makes such foundational tech hard to fund

Re: Claude Code daily benchmarks for degradation tracking

#298

Post by bushbaba »

levkk wrote: post #100
antirez wrote: post #29

Why I do not believe this shows Anthropic serves folks a worse model: 1. The percentage drop is too low and oscillating, it goes up and down. 2. The baseline of Sonnet 4.5 (the obvious choice for when they have GPU busy for the next training) should be established to see Opus at some point goes Sonnet level. This was not done but likely we would see a much sharp decline in certain days / periods. The graph would look…

I believe the science, but I've been using it daily and it's been getting worse, noticeably.

I’m finding Gemini and chatGPT web terminal to out perform Claude code. The context becomes too much for the LLM, and tries to make up for it by doing more file read ops.

Re: Claude Code daily benchmarks for degradation tracking

#300

Post by devonkelley »

Running agents in production, I've stopped trying to figure out why things degrade. The answer changes weekly.

Model drift, provider load, API changes, tool failures - it doesn't matter. What matters is that yesterday's 95% success rate is today's 70%, and by the time you notice, debug, and ship a fix, something else has shifted.

The real question isn't "is the model degraded?" It's "what should my agent do right now given current conditions?"

We ended up building systems that canary multiple execution paths continuously and route traffic based on what's actually working. When Claude degrades, traffic shifts to the backup path automatically. No alerts, no dashboards, no incident.

Treating this as a measurement problem assumes humans will act on the data. At scale, that assumption breaks.

Post reply on HN