Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

11–20 of 778 posts

Re: An update on recent Claude Code quality reports

#11

Post by teaearlgraycold »

> On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6.

Is it just me or does this seem kind of shocking? Such a severe bug affecting millions of users with a non-trivial effect on the context window that should be readily evident to anyone looking at the analytics. Makes me wonder if this is the result of Anthropic's vibe-coding culture. No one's actually looking at the product, its code, or its outputs?

Re: An update on recent Claude Code quality reports

#12

Post by Eridrus »

jryio wrote: post #2

1. They changed the default in March from high to medium, however Claude Code still showed high (took 1 month 3 days to notice and remediate) 2. Old sessions had the thinking tokens stripped, resuming the session made Claude stupid (took 15 days to notice and remediate) 3. System prompt to make Claude less verbose reducing coding quality (4 days - better) All this to say... the experience of suspecting a model is get…

To be fair to Anthropic, they did not intentionally degrade performance.

To take the opposite side, this is the quality of software you get atm when your org is all in on vibe coding everything.

Re: An update on recent Claude Code quality reports

#13

Post by mlinsey »

Alifatisk wrote: post #7

It’s incredible how forgiving you guys are with Anthropic and their errors. Especially considering you pay high price for their service and receive lower quality than expected.

The consumer surplus is quite high. Even with the regressions in this postmortem, performance was above the models last fall, when I was gladly paying for my subscription and thought it was net saving me time.

That said, there is now much better competition with Codex, so there's only so much rope they have now.

Re: An update on recent Claude Code quality reports

#15

Post by mannanj »

Robdel12 wrote: post #5

Wow, bad enough for them to actually publish something and not cryptic tweets from employees. Damage is done for me though. Even just one of these things (messing with adaptive thinking) is enough for me to not trust them anymore. And then their A/B testing this week on pricing.

so who do you trust and go to? (NotClearlySo)OpenAI?

Re: An update on recent Claude Code quality reports

#17

Post by qingcharles »

sroussey wrote: post #6
jryio wrote: post #2

1. They changed the default in March from high to medium, however Claude Code still showed high (took 1 month 3 days to notice and remediate) 2. Old sessions had the thinking tokens stripped, resuming the session made Claude stupid (took 15 days to notice and remediate) 3. System prompt to make Claude less verbose reducing coding quality (4 days - better) All this to say... the experience of suspecting a model is get…

None of these problems equate to degrading model performance. Completely different team. Degraded CC harness, sure.

Sure, but it gives the impression of degraded model performance. Especially when the interface is still saying the model is operating on "high", the same as it did yesterday, yet it is in "medium" -- it just looks like the model got hobbled.

Re: An update on recent Claude Code quality reports

#18

Post by manmal »

teaearlgraycold wrote: post #11

> On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6. Is it just me or does this seem kind o…

I think that would also have busted cache all the time, and uncached requests consume usage limits rapidly.

Re: An update on recent Claude Code quality reports

#19

Post by xlayn »

If anthropic is doing this as a result of "optimizations" they need to stop doing that and raise the price. The other thing, there should be a way to test a model and validate that the model is answering exactly the same each time. I have experienced twice... when a new model is going to come out... the quality of the top dog one starts going down... and bam.. the new model is so good.... like the previous one 3 months ago.

The other thing, when anthropic turns on lazy claude... (I want to coin here the term Claudez for the version of claude that's lazy.. Claude zzZZzz = Claudez) that thing is terrible... you ask the model for something... and it's like... oh yes, that will probably depend on memory bandwith... do you want me to search that?...

YES... DO IT... FRICKING MACHINE..

Post reply on HN