Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

Post reply on HN
Print view View original

261–270 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#261

Post by F7F7F7 »

eterm wrote: post #76
antirez wrote: post #29

Why I do not believe this shows Anthropic serves folks a worse model: 1. The percentage drop is too low and oscillating, it goes up and down. 2. The baseline of Sonnet 4.5 (the obvious choice for when they have GPU busy for the next training) should be established to see Opus at some point goes Sonnet level. This was not done but likely we would see a much sharp decline in certain days / periods. The graph would look…

4. The graph starts January 8. Why January 8? Was that an outlier high point? IIRC, Opus 4.5 was released late november.

Right after the Holiday double token promotion users felt (perceived) a huge regression in capabilities. I bet that triggered the idea.

Re: Claude Code daily benchmarks for degradation tracking

#262

Post by F7F7F7 »

Dowwie wrote: post #12

Simply search user prompts for curse words and then measure hostility sentiment. User hostility rises as agents fail to meet expectations.

There’s a correlation between getting the “How’s Claude Doing This Session?” (Or whatever) and four letter words.

It’s not always then, but it often follows it.

Re: Claude Code daily benchmarks for degradation tracking

#265

Post by justinlivi »

conception wrote: post #209
davidee wrote: post #169

Earlier quoted context omitted.

I have to concur. And to the question about understanding what its good and bad at; no, tasks that it could accomplish quickly and easily just a month ago, now require more detailed prompting and constant "erroneous direction correction." It's almost as if, as tool use and planning capabilities have expanded, Claude (as a singular product) is having a harder time coming up with simple approaches that just work, inste…

I assume, after any compacting of the context window that the session is more or less useless at that point I’ve never had consistent results after compacting.

Compacting equals death of the session in my process. I do everything I can to avoid hitting it. If I accidentally fly too close to the sun and compact I tend to revert and start fresh. As soon as it compacts it's basically useless

Re: Claude Code daily benchmarks for degradation tracking

#266

Post by minimaltom »

bonoboTP wrote: post #248
minimaltom wrote: post #162

Earlier quoted context omitted.

Its not deterministic. Any individual floating point mul/add is deterministic, but in a GPU these are all happening in parallel and the accumulation is in the order they happen to complete. When you add A then B then C, you get a different answer than C then A then B, because floating point, approximation error, subnormals etc.

It can be made deterministic. It's not trivial and can slow it down a bit (not much) but there are environment variables you can set to make your GPU computations bitwise reproducible. I have done this in training models with Pytorch.

There are settings to make it reproducible but they incur a non-negligible drop in performance.

Unsurprising given they amount to explicit synchronization to make the order of operations deterministic.

Re: Claude Code daily benchmarks for degradation tracking

#267

Post by cma »

trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

For the models themselves, less so for the scaffolding, considering things like the long running TPU bug that happened, are there not internal quality measures looking at samples of real outputs? Using the real systems on benchmarks and looking for degraded perf or things like skipping refusals? Aside from degrading stuff for users, with the focus on AI safety wouldn't that be important to have in case an inference bug messes with something that affects the post training and it starts giving out dangerous bioweapon construction info or the other things that are guarded against and talked about in the model cards?

Re: Claude Code daily benchmarks for degradation tracking

#268

Post by simsla »

nikcub wrote: post #253
ofirpress wrote: post #24

[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.

> I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. assume this is because of model costs. anthropic could either throw some credits their way (would be worthwhile to dispel the 80 reddit posts a day about degrading models and quantization) or OP could throw up a donation / tip link

Probably, but with a small sample size like that, they should probably be taking the uncertainty into account, because I wouldn't be surprised if a lot of this variation falls within expected noise.

E.g. some binomial interval proportions (aka confidence intervals).

Re: Claude Code daily benchmarks for degradation tracking

#269

Post by samlinnfer »

trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Is there compensation for the tokens because Claude wasted all of them?

Re: Claude Code daily benchmarks for degradation tracking

#270

Post by jonaustin »

trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

How about how Claude 2.1.x is "literally unusable" because it frequently completely hangs (requires kill -9) and uses 100% cpu?

https://github.com/anthropics/claude-code/issues/18532

Post reply on HN