Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

451–460 of 778 posts

Re: An update on recent Claude Code quality reports

#451

Post by kiratp »

CjHuber wrote: post #187
btown wrote: post #169

Earlier quoted context omitted.

Is there a way to say: I am happy to pay a premium (in tokens or extra usage) to make sure that my resumed 1h+ session has all the old thinking? I understand you wouldn't want this to be the default, particularly for people who have one giant running session for many topics - and I can only imagine the load involved in full cache misses at scale. But there are other use cases where this thinking is critical - for ins…

I think it’s crazy that they do this, especially without any notice. I would not have renewed my subscription if I knew that they started doing this. Especially in the analysis part of my work I don‘t care about the actual text output itself most of the time but try to make the model „understand“ the topic. In the first phase the actual text output itself is worthless it just serves as an indicator that the context w…

OpenAI does this for all API calls

> Our systems will smartly ignore any reasoning items that aren’t relevant to your functions, and only retain those in context that are relevant. You can pass reasoning items from previous responses either using the previous_response_id parameter, or by manually passing in all the output items from a past response into the input of a new one.

https://developers.openai.com/api/docs/guides/reasoning

Disclosure - work on AI@msft

Re: An update on recent Claude Code quality reports

#452

Post by raincole »

tadfisher wrote: post #110
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

It astounds me that a company valued in the hundreds-of-billions-of-dollars has written this. One of the following must be true: 1. They actually believed latency reduction was worth compromising output quality for sessions that have already been long idle. Moreover, they thought doing so was better than showing a loading indicator or some other means of communicating to the user that context is being loaded. 2. What…

It's very weird that they frame caching as "latency reduction" when it comes to a cloud service. I mean, yes, technically it reduces latency, but more importantly it reduces cost. Sometimes it's more than 80% of the total cost.

I'm sure most companies and customers will consider compromising quality for 80% cost reduction. If they just be honest they'll be fine.

Re: An update on recent Claude Code quality reports

#454

Post by kiratp »

trinsic2 wrote: post #413
bobkb wrote: post #220

Earlier quoted context omitted.

Resuming sessions after more than 1 hour is a very common workflow that many teams are following. It will be great if this is considered as an expected behaviour and design the UX around it. Perhaps you are not realising the fact that Claude code has replaced the shells people were using (ie now bash is replaced with a Claude code session).

I think thats a bad idea. It seems like expecting to have a prompt open like this, accumulating context puts a load on the back end. Its one of those things that is a bad habit. Like trying to maintain open tabs in a browser as a way to keep your work flow up to date when what you really should be doing is taking notes of your process and working from there. I have project folders/files and memory stored for each ses…

Agents making forward progress hours apart is an expected pattern and inference engines are being adapted to serve that purpose well.

It’s hard to do it without killing performance and requires engineering in the DC to have fast access to SSDs etc.

Disclosure: work on ai@msft. Opinions my own.

Re: An update on recent Claude Code quality reports

#455

Post by switchbak »

nixpulvis wrote: post #356
esafak wrote: post #332

Earlier quoted context omitted.

They have to know that this could bite them and to ask the question first.

I do think having some insight into the current state of the cache and a realistic estimate for prompt token use is something we should demand.

If there was an affordance on the TUI that made this visible and encouraged users to learn more - that would go a long way.

Re: An update on recent Claude Code quality reports

#456

Post by xpe »

thomassmith65 wrote: post #446
epsteingpt wrote: post #425

Earlier quoted context omitted.

They gaslit people for months saying it wasn't an issue publicly. That's the reason for the flak

And still are gaslighting: We take reports about degradation very seriously. We never intentionally degrade our models [...] On March 4, we changed Claude Code's default reasoning effort from high to medium Anthropic is the best company of its kind, but that is badly worded PR.

To my eye, gaslighting is a serious accusation. Wikipedia's first line matches how I think of it: "Gaslighting is the manipulation of someone into questioning their perception of reality."

Did I miss something? I'm only looking at primary sources to start. Not Reddit. Not The Register. Official company communications.

Did Anthropic tell users i.e. "you are wrong, your experience is not worse."? If so, that would reach the bar of gaslighting, as I understand it (and I'm not alone). If you have a different understanding, please share what it is so I understand what you mean.

Re: An update on recent Claude Code quality reports

#457

Post by jorjon »

bcherny wrote: post #124
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

What about:

/loop 5m say "ok".

Will that keep the cache fresh?

Re: An update on recent Claude Code quality reports

#459

Post by esafak »

elAhmo wrote: post #219
cedws wrote: post #160

>On April 16, we added a system prompt instruction to reduce verbosity In practice I understand this would be difficult but I feel like the system prompt should be versioned alongside the model. Changing the system prompt out from underneath users when you've published benchmarks using an older system prompt feels deceptive. At least tell users when the system prompt has changed.

Its also kinda funny they have to rely on system prompt to control verbosity itself.

It's cheaper than retraining the model.

Re: An update on recent Claude Code quality reports

#460

Post by Eridrus »

shrx wrote: post #287
Eridrus wrote: post #12

Earlier quoted context omitted.

To be fair to Anthropic, they did not intentionally degrade performance. To take the opposite side, this is the quality of software you get atm when your org is all in on vibe coding everything.

Are you saying dropping cache after 1 hour is not intentionally degrading performance?

Yes. Caching is a cost optimization not a response quality metric.
Post reply on HN