Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

Post reply on HN
Print view View original

251–260 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#251

Post by mrandish »

pluralmonad wrote: post #244
mrandish wrote: post #235

Earlier quoted context omitted.

> these are earnest people. I agree. I'll also add that when my startup got acquired into a very large, well-known valley giant with a sterling rep for integrity and I ended up as a senior executive - over time I got a first-hand education on the myriad ways genuinely well-intentioned people can still end up being the responsible party(s) presiding over a system doing net-wrong things. All with no individual ever mea…

Someone posted this here the other day and it uses _Demons_ to discuss exactly your point. https://possessedmachines.com/

Wow. Only one page in and already bookmarked to absorb later. Thanks for the link.

Re: Claude Code daily benchmarks for degradation tracking

#253

Post by nikcub »

ofirpress wrote: post #24

[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.

> I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score.

assume this is because of model costs. anthropic could either throw some credits their way (would be worthwhile to dispel the 80 reddit posts a day about degrading models and quantization) or OP could throw up a donation / tip link

Re: Claude Code daily benchmarks for degradation tracking

#254

Post by F7F7F7 »

Aurornis wrote: post #146
bgirard wrote: post #120

Earlier quoted context omitted.

> malicious It doesn't have to be malicious. If my workflow is to send a prompt once and hopefully accept the result, then degradation matters a lot. If degradation is causing me to silently get worse code output on some of my commits it matters to me. I care about -expected- performance when picking which model to use, not optimal benchmark performance.

Non-determinism isn’t the same as degradation. The non-determinism means that even with a temperature of 0.0, you can’t expect the outputs to be the same across API calls. In practice people tend to index to the best results they’ve experienced and view anything else as degradation. In practice it may just be randomness in either direction from the prompts. When you’re getting good results you assume it’s normal. Whe…

“Just drink the water, it’s all water.”

Re: Claude Code daily benchmarks for degradation tracking

#255

Post by phist_mcgee »

nikcub wrote: post #253
ofirpress wrote: post #24

[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.

> I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. assume this is because of model costs. anthropic could either throw some credits their way (would be worthwhile to dispel the 80 reddit posts a day about degrading models and quantization) or OP could throw up a donation / tip link

Then you'd get people claiming that the benchmarks were 'paid for' by anthropic

Re: Claude Code daily benchmarks for degradation tracking

#256

Post by F7F7F7 »

davidee wrote: post #169
levkk wrote: post #100

Earlier quoted context omitted.

I believe the science, but I've been using it daily and it's been getting worse, noticeably.

I have to concur. And to the question about understanding what its good and bad at; no, tasks that it could accomplish quickly and easily just a month ago, now require more detailed prompting and constant "erroneous direction correction." It's almost as if, as tool use and planning capabilities have expanded, Claude (as a singular product) is having a harder time coming up with simple approaches that just work, inste…

Multiple concurrences a choir or a mob?

1pm EST time it’s all down hill until around 8 or 9pm EST time.

Late nights and weekends is smooth sailing.

Re: Claude Code daily benchmarks for degradation tracking

#257

Post by nikcub »

phist_mcgee wrote: post #255
nikcub wrote: post #253

Earlier quoted context omitted.

> I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. assume this is because of model costs. anthropic could either throw some credits their way (would be worthwhile to dispel the 80 reddit posts a day about degrading models and quantization) or OP could throw up a donation / tip link

Then you'd get people claiming that the benchmarks were 'paid for' by anthropic

one thing you learn from being on the internet is that you're never going to satisfy everybody

Re: Claude Code daily benchmarks for degradation tracking

#258

Post by jonawesomegreen »

I’ve noticed Claude has been noticeably worse over the last week. For example, it told me I should pass frozen to make my Enum immutable—that’s not a thing. (It is a thing for dataclasses, but not for Enums.) That’s a pretty basic language feature it was nailing until recently. It also suggested I parse a URL using urlparse in a function that already uses urlparse. These are basic mistakes it wasn’t making before. Something seems to have changed, but I’m not sure what.

Re: Claude Code daily benchmarks for degradation tracking

#259

Post by F7F7F7 »

warkdarrior wrote: post #104
levkk wrote: post #100

Earlier quoted context omitted.

I believe the science, but I've been using it daily and it's been getting worse, noticeably.

Is it possible that your expectations are increasing, not that the model is getting worse?

I’ve had Opus struggle on trivial things that Sonnet 3.5 handled with ease.

It’s not so much that the implementations are bad because the code is bad (the code is bad). It’s that it gets extremely confused and starts to frantically make worse and worse decisions and questioning itself. Editing multiple files, changing its mind and only fixing one or two. Reseting and overriding multiple batches of commits without so much as a second thought and losing days of work (yes, I’ve learned my lesson).

It, the model, can’t even reason with the decisions it’s making from turn to turn. And the more opaque agentic help it’s getting the more I suspect that tasks are being routed to much lesser models (not the ones we’ve chosen via /model or those in our agent definitions) however Anthropic chooses.

In these moments I mind as well be using Haiku.

Re: Claude Code daily benchmarks for degradation tracking

#260

Post by F7F7F7 »

TIPSIO wrote: post #177
antirez wrote: post #29

Why I do not believe this shows Anthropic serves folks a worse model: 1. The percentage drop is too low and oscillating, it goes up and down. 2. The baseline of Sonnet 4.5 (the obvious choice for when they have GPU busy for the next training) should be established to see Opus at some point goes Sonnet level. This was not done but likely we would see a much sharp decline in certain days / periods. The graph would look…

I too suspect the A/B testing is the prime suspect: context window limits, system prompts, MAYBE some other questionable things that should be disclosed. Either way, if true, given the cost I wish I could opt-out or it were more transparent. Put out variants you can select and see which one people flock to. I and many others would probably test constantly and provide detailed feedback. All speculation though

Whenever I see new behaviors and suspect I’m being tested on I’ll typically see a feedback form at some point in that session. Well, that and dropping four letter words.

I know it’s more random sampling than not. But they are definitely using our codebases (and in some respects our livelihoods) as their guinea pigs.

Post reply on HN