Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Claude Opus performance affected by time of day?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Ask HN: Claude Opus performance affected by time of day?

#31

Post by jdcasale »

The math is obvious on this one. It's super well-documented that model performance on complex tasks scales (to some asymptote) with the amount of inference-time compute allocated.

LLM providers must dynamically scale inference-time compute based on current load because they have limited compute. Thus it's impossible for traffic spikes _not_ to cause some degradations in model performance (at least until/unless they acquire enough compute to saturate that asymptotic curve for every request under all demand conditions -- it does not seem plausible that they are anywhere close to this)

Re: Ask HN: Claude Opus performance affected by time of day?

#32

Post by pankajdoharey »

bayarearefugee wrote: post #2

I mostly use Gemini, so I can't speak for Claude, but Gemini definitely has variable quality at different times, though I've never bothered to try to find a specific time-of-day pattern to it. The most reliable time to see it fall apart is when Google makes a public announcement that is likely to cause a sudden influx of people using it. And there are multiple levels of failure, first you start seeing iffy responses…

Its the router they are using, we surely are not getting what we select. Also after a few queries the intelligence drops. abruptly. and doesn't recover even after we start a new session, so there is another internal quota at play.

Re: Ask HN: Claude Opus performance affected by time of day?

#33

Post by pankajdoharey »

astrange wrote: post #24
kevinsync wrote: post #5

Earlier quoted context omitted.

I suspect they might transparently fall back too; Opus 4.5 has been really reasonable lately, except right after it launched, and also surrounding any service interruptions / problems reported on status.claude.ai -- once those issues resolve, for a few hours the results feel very "Sonnet", and it starts making a lot more mistakes. When that happens, I'll usually just pause Claude and prompt Codex and Gemini with the…

They don't ever fall back to cheaper models silently. What Anthropic does do is poke the model to tell you to go to bed if you use it too long ("long conversation reminder") which distracts it from actually answering. Sometimes they do have associations with things like the day of the year and might be lazier some months than others.

If they are real slime balls they can justify it by saying you see we use speculative decoding so we first use a smaller faster model model first and then then answer is enhanced by larger model blah blah ..... "FOr the best User experience"

Re: Ask HN: Claude Opus performance affected by time of day?

#34

Post by pankajdoharey »

stavros wrote: post #20
kevinsync wrote: post #5

Earlier quoted context omitted.

I suspect they might transparently fall back too; Opus 4.5 has been really reasonable lately, except right after it launched, and also surrounding any service interruptions / problems reported on status.claude.ai -- once those issues resolve, for a few hours the results feel very "Sonnet", and it starts making a lot more mistakes. When that happens, I'll usually just pause Claude and prompt Codex and Gemini with the…

I didn't believe such conspiracy theories, until one day I noticed Sonnet 4.5 (which I had been using for weeks to great success) perform much worse, very visibly so. A few hours later, Opus 4.5 was released. Now I don't know what to think.

Model router.

Re: Ask HN: Claude Opus performance affected by time of day?

#36

Post by YetAnotherNick »

jdcasale wrote: post #31

The math is obvious on this one. It's super well-documented that model performance on complex tasks scales (to some asymptote) with the amount of inference-time compute allocated. LLM providers must dynamically scale inference-time compute based on current load because they have limited compute. Thus it's impossible for traffic spikes _not_ to cause some degradations in model performance (at least until/unless they a…

Umm. I run multiple benchmark using APIs for my work and the inference time compute allotted has clear correlation with the metrics. But time of the day certainly isn't. If it is that straightforward people can prove very easily rather than relying on the anecdotes.

They either overprovision the server during low demand or they might dynamically provision servers based on load.

Re: Ask HN: Claude Opus performance affected by time of day?

#37

Post by SOLAR_FIELDS »

YetAnotherNick wrote: post #36
jdcasale wrote: post #31

The math is obvious on this one. It's super well-documented that model performance on complex tasks scales (to some asymptote) with the amount of inference-time compute allocated. LLM providers must dynamically scale inference-time compute based on current load because they have limited compute. Thus it's impossible for traffic spikes _not_ to cause some degradations in model performance (at least until/unless they a…

Umm. I run multiple benchmark using APIs for my work and the inference time compute allotted has clear correlation with the metrics. But time of the day certainly isn't. If it is that straightforward people can prove very easily rather than relying on the anecdotes. They either overprovision the server during low demand or they might dynamically provision servers based on load.

Yes, every time I see some variant of this come up (and believe me, this has been coming up since before the GPT3.5 days) there’s never any actual data demonstrating that it’s the case. As you say, it should be completely trivial to run the exact same prompt multiple times per day and capture the output to demonstrate this.

But no one ever seems to do that, they are rather content to “feel” that this is the case instead

Re: Ask HN: Claude Opus performance affected by time of day?

#38

Post by 8note »

anonzzzies wrote: post #8

Many people 'notice' it (on reddit); I notice it too, but it is hard to prove. I tried the same prompt on the same code every 4 hours for 48 hours, the behaviour was slightly different but not worse or much different in time. But then I just work on my normal code, think wtf is it doing now??? look at the time and see it is US day time and stop. People put forward many theories for this (weaker model routing; be it a…

you might want to capture all your context when its doing badly?

maybe youre operating at very full, or with some poison pill file?

keep some logs of when cc reads different files vs when it gets stupid?

Re: Ask HN: Claude Opus performance affected by time of day?

#39

Post by cma »

dcre wrote: post #25
denysvitali wrote: post #22

Earlier quoted context omitted.

They did yes, but should we trust them? I remember clearly this problem happening in the past, despite their claims. I initially thought it was an elaborate hoax, but it turned out to be factually true in my case.

I tend to think it would be very hard and very risky for large, successful companies to systematically lie about these things without getting caught, and the people who would be doing the lying in this case are not professional liars, they’re engineers who generally seem trustworthy. So yes, if there is a degradation, I think bugs are much more likely than systematic lying.

The TPU implementation used approximate top-k instead of the exact used on nvidia. While that wouldn't matter too much and there was a bug with it, it still was a cost savings thing not to use exact from the beginning because it wasn't efficient on TPUs which they were routing to under load. So it was a bit of a model difference under load, even aside from the bug.

Re: Ask HN: Claude Opus performance affected by time of day?

#40

Post by dcre »

cma wrote: post #39
dcre wrote: post #25

Earlier quoted context omitted.

I tend to think it would be very hard and very risky for large, successful companies to systematically lie about these things without getting caught, and the people who would be doing the lying in this case are not professional liars, they’re engineers who generally seem trustworthy. So yes, if there is a degradation, I think bugs are much more likely than systematic lying.

The TPU implementation used approximate top-k instead of the exact used on nvidia. While that wouldn't matter too much and there was a bug with it, it still was a cost savings thing not to use exact from the beginning because it wasn't efficient on TPUs which they were routing to under load. So it was a bit of a model difference under load, even aside from the bug.

To the extent this is an accurate characterization (somewhat, I think), they considered the quality difference a bug and fixed it!
Post reply on HN