Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

651–660 of 778 posts

Re: An update on recent Claude Code quality reports

#651

Post by cowlby »

bcherny wrote: post #124
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

Ahh that makes sense. Sometimes it's convenient to re-use an older conversation that has all the context I need. But maybe it's just the last 20% that's relevant.

It would be nice to be able to summarize/cut into a new leaner conversation vs having to coax all the context back into a fresh one. Something like keep the last 100,000 tokens.

I believe /compact achieves something like this? It just takes so long to summarize that it creates friction.

Re: An update on recent Claude Code quality reports

#652

Post by Gareth321 »

time0ut wrote: post #647
podnami wrote: post #77

They lost me at Opus 4.7 Anecdotally OpenAI is trying to get into our enterprise tooth and nail, and have offered unlimited tokens until summer. Gave GPT5.4 a try because of this and honestly I don’t know if we are getting some extra treatment, but running it at extra high effort the last 30 days I’ve barely see it make any mistakes. At some points even the reasoning traces brought a smile to my face as it preemptive…

Opus 4.7 via code has been inconsistent for me. Sometimes, it feels like working with a brilliant collaborator and is as good as 4.5 and 4.6 were. Other times, it takes dumb and lazy short cuts. It can be quite frustrating. Its response when I tell it it did something wrong is often to write a memory... which is then does not always read. The inconsistency isn't due to session length or age either. These are all new…

My experience as well. This is even worse than just having a mediocre model, because I can work around that. The inconsistency means it produces different outputs for the same prompt, and I can't rely on that as a business tool.

Re: An update on recent Claude Code quality reports

#653

Post by mtilsted »

dcchambers wrote: post #237

So it turns out Anthropic was gaslighting everyone on twitter about this then? Swearing that nothing had changed and people were imagining the models got worse?

Nope, they were technical correct. Nothing had changed with the model. The model had not gotten any worse.

The harness on the other hand. Now that had problems.

Re: An update on recent Claude Code quality reports

#654

Post by airstrike »

bede wrote: post #601
dbeardsl wrote: post #162

Earlier quoted context omitted.

I appreciate the reply, but I was never under the impression that gaps in conversations would increase costs nor reduce quality. Both are surprising and disappointing. I feel like that is a choice best left up to users. i.e. "Resuming this conversation with full context will consume X% of your 5-hour usage bucket, but that can be reduced by Y% by dropping old thinking logs"

I too would far rather bear a token cost than have my sessions rot silently beneath my feet. I usually have ~5 running CC sessions, some of which I may leave for a week or two of inactivity at a time.

same here, and I suspect there are dozens of us

Re: An update on recent Claude Code quality reports

#655

Post by whalesalad »

hansmayer wrote: post #614

A suggestion to Anthropic, just start charging the real price for your software. Of course you have to dumb it down, when the $200 tier in reality produces 5-10 thousand dollars in monthly costs when used by people who know how to max it out. So then you come up with creative nonsense like "adaptive thinking" when your tool is sometimes working and sometimes outright not - the irony of "intelligent tools" not "thinki…

> when the $200 tier in reality produces 5-10 thousand dollars in monthly costs

are you asserting that the actual dollar cost to anthropic for a heavy user was 5-10k? or are you basing this on the (fabricated) value of those tokens, ie potentially lost revenue from a pay-per-token user.

Re: An update on recent Claude Code quality reports

#656

Post by Aperocky »

Bishonen88 wrote: post #521
Aperocky wrote: post #480

Earlier quoted context omitted.

Wow, I always thought the context is always stored locally and this is something I have control over. Glad I use kiro-cli which doesn't do this.

you might be biased due to your employment :)

Objectively speaking, I want control of context and when I compact it.

That wouldn't change with employment.

Re: An update on recent Claude Code quality reports

#657

Post by YetAnotherNick »

Why don't they monitor average prompt and response token length(both cached and uncached) per interaction. Seems this could have solved all their previous unnoticed degradation.

Also bit surprised they don't have any automated quality check. They can run something like swe bench before each release. Both of these seem like a basic thing even for startup, let alone some product generating billions in revenue.

Re: An update on recent Claude Code quality reports

#658

Post by dgreensp »

This reveals a staggering level of incompetence, if that’s really all it is, and lack of transparency.

They don’t have ANY product-level quality tests that picked this up? Many users did their own tests and published them. It’s not hard. And these users’ complaints were initially dismissed.

I don’t think the high vs medium change is really on par with the others. That’s a setting you change in the UI, and depending on what you are doing, both effort levels are pretty capable, they just operate a bit differently. Unless I’m missing something and they are saying they were doing some kind of routing behind the scenes.

If they are constantly pushing major changes to the prompts and workings of the tool, without communicating about it, and without testing, it’s likely there are other bugs and quality-degrading changes beyond the ones in this article, which would make a lot of sense.

Re: An update on recent Claude Code quality reports

#659

Post by [deleted] »

danso wrote: post #461
sharts wrote: post #382

Earlier quoted context omitted.

That doesn’t make sense to pay more for cache warming. Your session for the most part is already persisted. Why would it be reasonable to pay again to continue where you left off at any time in the future?

Genuine question: is the cost to keep a persistent warmed cache for sessions idling for hours/days not significant when done for hundreds of thousands of users? Wouldn’t it pose a resource constraint on Anthropic at some point?

[deleted]

Re: An update on recent Claude Code quality reports

#660

Post by kolinko »

dgreensp wrote: post #658

This reveals a staggering level of incompetence, if that’s really all it is, and lack of transparency. They don’t have ANY product-level quality tests that picked this up? Many users did their own tests and published them. It’s not hard. And these users’ complaints were initially dismissed. I don’t think the high vs medium change is really on par with the others. That’s a setting you change in the UI, and depending o…

There were few systems like claude in the past, to testing rulebook is not really written yet. And far from obvious.
Post reply on HN