Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

Post reply on HN
Print view View original

111–120 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#111

Post by eli »

dajonker wrote: post #21

Wouldn't be surprised if they slowly start quantizing their models over time. Makes it easier to scale and reduce operational cost. Also makes a new release have more impact as it will be more notably "better" than what you've been using the past couple of days/weeks.

I would be surprised tbh.

Anthropic does not exactly act like they're constrained by infra costs in other areas, and noticeably degrading a product when you're in tight competition with 1 or 2 other players with similar products seems like a bad place to start.

I think people just notice the flaws in these models more the longer they use them. Aka the "honeymoon-hangover effect," a real pattern that has been shown in a variety of real world situations.

Re: Claude Code daily benchmarks for degradation tracking

#112

Post by DanielHall »

jampa wrote: post #57

I am using API mode, and it's clear that there are times when the Claude model just gives up. And it is very noticeable because the model just does the most dumb things possible. "You have a bug in line 23." "Oh yes, this solution is bugged, let me delete the whole feature." That one-line fix I could make even with ChatGPT 3.5 can't just happen. Workflows that I use and are very reproducible start to flake and then f…

I encountered the same situation too; Claude has 'become lazy'.

Re: Claude Code daily benchmarks for degradation tracking

#113

Post by GoatInGrey »

seunosewa wrote: post #80
ofirpress wrote: post #24

[SWE-bench co-author here] It seems like they run this test on a subset of 50 tasks, and that they only run the test once per day. So a lot of the movement in accuracy could be attributed to that. I would run on 300 tasks and I'd run the test suite 5 or 10 times per day and average that score. Lots of variance in the score can come from random stuff like even Anthropic's servers being overloaded.

The degradation may be more significant within the day than at the same time every day.

Sure, but it's still useful insight to see how it performs over time. Of course, cynically, Anthropic could game the benchmark by routing this benchmark's specific prompts to an unadulterated instance of the model.

Re: Claude Code daily benchmarks for degradation tracking

#114

Post by dmos62 »

Lack of transparency as regards "thinking power"-consistency is a big gripe of mine with LLM providers. It's even worse with ChatGPT and the like. E.g. I had to learn the hard way that at >45k input tokens ChatGPT 5.2 Thinking Extended bumps its intelligence down so hard that it can't follow basic instructions (or it somehow truncates the input, losing the instructions). It sucks to lose confidence in an otherwise great tool. I would 100x prefer being forced to back-off, or getting a straight-no, than getting silently downgraded. Transparency is a big deal.

Re: Claude Code daily benchmarks for degradation tracking

#116

Post by cmrdporcupine »

TedDallas wrote: post #99
codeflo wrote: post #79

Earlier quoted context omitted.

This is intentional? I think delivering lower quality than what was advertised and benchmarked is borderline fraud, but YMMV.

Per Anthropic’s RCA linked in Ops post for September 2025 issues: “… To state it plainly: We never reduce model quality due to demand, time of day, or server load. …” So according to Anthropic they are not tweaking quality setting due to demand.

I guess I just don't know how to square that with my actual experiences then.

I've seen sporadic drops in reasoning skills that made me feel like it was January 2025, not 2026 ... inconsistent.

Re: Claude Code daily benchmarks for degradation tracking

#117

Post by plagiarist »

epolanski wrote: post #47
ofirpress wrote: post #32

Earlier quoted context omitted.

Benchmarks can get costly to run- you can reach out to frontier model creators to try and get them to give you free credits, but usually they'll only agree to that once your benchmark is pretty popular.

The last thing a proper benchmark should do is reveal it's own API key.

IMO it should need a third party running the LLM anyway. Otherwise the evaluated company could notice they're receiving the same requests daily and discover benchmarking that way.

Re: Claude Code daily benchmarks for degradation tracking

#118

Post by GoatInGrey »

warkdarrior wrote: post #104
levkk wrote: post #100

Earlier quoted context omitted.

I believe the science, but I've been using it daily and it's been getting worse, noticeably.

Is it possible that your expectations are increasing, not that the model is getting worse?

Possible, though you eventually run into types of issues that you recall the model just not having before. Like accessing a database or not following the SOP you have it read each time it performs X routine task. There are also patterns that are much less ambiguous like getting caught in loops or failing to execute a script it wrote after ten attempts.

Re: Claude Code daily benchmarks for degradation tracking

#119

Post by kingstnap »

chrisjj wrote: post #92
vidarh wrote: post #60

Earlier quoted context omitted.

It would happen if they quietly decide to serve up more aggressively distilled / quantised / smaller models when under load.

They advertise the Opus 4.5 model. Secretly substituting a cheaper one to save costs would be fraud.

Old school Gemini used to do this. It was super obvious because mid day the model would go from stupid to completely brain dead. I have a screenshot of Google's FAQ on my PC from 2024-09-13 that says this (I took it to post to discord):

> How do I know which model Gemini is using in its responses?

> We believe in using the right model for the right task. We use various models at hand for specific tasks based on what we think will provide the best experience.

Re: Claude Code daily benchmarks for degradation tracking

#120

Post by bgirard »

botacode wrote: post #107
Davidzheng wrote: post #43

Earlier quoted context omitted.

but degradation from servers being overloaded would be the type of degradation this SHOULD measure no? Unless it's only intended for measuring their quietly distilling models (which they claim not to do? idk for certain)

Load just makes LLMs behave less deterministically and likely degrade. See: https://thinkingmachines.ai/blog/defeating-nondeterminism-in... They don't have to be malicious operators in this case. It just happens.

> malicious

It doesn't have to be malicious. If my workflow is to send a prompt once and hopefully accept the result, then degradation matters a lot. If degradation is causing me to silently get worse code output on some of my commits it matters to me.

I care about -expected- performance when picking which model to use, not optimal benchmark performance.

Post reply on HN