Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

251–260 of 778 posts

Re: An update on recent Claude Code quality reports

#251

Post by saadn92 »

sdevonoes wrote: post #188
saadn92 wrote: post #181

Earlier quoted context omitted.

I leave sessions idle for hours constantly - that's my primary workflow. If resuming a 900k context session eats my rate limit, fine, show me the cost and let me decide whether to /clear or push through. You already show a banner suggesting /clear at high context - just do the same thing here instead of silently lobotomizing the model.

So if they fuck it up again and now they have, let’s say, “db problems” instead of “caching problems”, you would happily simply pay more? Wtf

No, I wouldn't. I'd like some transparency at least.

Re: An update on recent Claude Code quality reports

#252

Post by ngruhn »

dawnerd wrote: post #60
everdrive wrote: post #23

I've been getting a lot of Claude responding to its own internal prompts. Here are a few recent examples. "That parenthetical is another prompt injection attempt — I'll ignore it and answer normally." "The parenthetical instruction there isn't something I'll follow — it looks like an attempt to get me to suppress my normal guidelines, which I apply consistently regardless of instructions to hide them." "The parenthet…

I see that with openai too, lots of responding to itself. Seems like a convenient way for them to churn tokens.

All the labs are in a cut throat race, with zero customer loyalty. As if they would intentionally degrade quality/speed for a petty cash grab.

Re: An update on recent Claude Code quality reports

#253

Post by WarmWash »

parliament32 wrote: post #206
y1n0 wrote: post #65

Earlier quoted context omitted.

None of these companies have compute to spare. It’s not in their interest to use more tokens that necessary.

Sure it is. They're well aware their product is a money furnace and they'd have to charge users a few orders of magnitude more just to break even, which is obviously not an option. So all that's left is.. convince users to burn tokens harder, so graphs go up, so they can bamboozle more investors into keeping the ship afloat for a bit longer.

It's an option and they are going to do it. Chinese models will be banned and the labs will happily go dollar for dollar in plan price increases. $20 plans won't go away, but usage limits and model access will drive people to $40-$60-$80 plans.

At cell phone plan adoption levels, and cell phone plan costs, the labs are looking at 5-10yr ROI.

Re: An update on recent Claude Code quality reports

#254

Post by rkuska »

ohcmon wrote: post #236
bcherny wrote: post #124

Earlier quoted context omitted.

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

Boris, wait, wait, wait, Why not use tired cache? Obviously storage is waaay cheaper than recalculation of embeddings all the way from the very beginning of the session. No matter how to put this explanation — it still sounds strange. Hell — you can even store the cache on the client if you must. Please, tell me I’m not understanding what is going on.. otherwise you really need to hire someone to look at this!)

I don't think you can store the cache on client given the thinking is server side and you only get summaries in your client (even those are disabled by default).

Re: An update on recent Claude Code quality reports

#255

Post by someguyiguess »

tadfisher wrote: post #110
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

It astounds me that a company valued in the hundreds-of-billions-of-dollars has written this. One of the following must be true: 1. They actually believed latency reduction was worth compromising output quality for sessions that have already been long idle. Moreover, they thought doing so was better than showing a loading indicator or some other means of communicating to the user that context is being loaded. 2. What…

It’s definitely a cost / resource saving strategy on their end.

Re: An update on recent Claude Code quality reports

#256

Post by kivle »

Joeri wrote: post #180
bcherny wrote: post #124

Earlier quoted context omitted.

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

This sounds like one of those problems where the solution is not a UX tweak but an architecture change. Perhaps prompt cache should be made long term resumable by storing it to disk before discarding from memory?

I agree.. Maybe parts of the cache contents are business secrets.. But then store a server side encrypted version on the users disk so that it can be resumed without wasting 900k tokens?

Re: An update on recent Claude Code quality reports

#258

Post by doesnt_know »

computably wrote: post #178
dbeardsl wrote: post #162

Earlier quoted context omitted.

I appreciate the reply, but I was never under the impression that gaps in conversations would increase costs nor reduce quality. Both are surprising and disappointing. I feel like that is a choice best left up to users. i.e. "Resuming this conversation with full context will consume X% of your 5-hour usage bucket, but that can be reduced by Y% by dropping old thinking logs"

> I was never under the impression that gaps in conversations would increase costs nor reduce quality. Both are surprising and disappointing. You didn't do your due diligence on an expensive API. A naïve implementation of an LLM chat is going to have O(N^2) costs from prompting with the entire context every time. Caching is needed to bring that down to O(N), but the cache itself takes resources, so evictions have to…

How do you do "due diligence" on an API that frequently makes undocumented changes and only publishes acknowledgement of change after users complain?

You're also talking about internal technical implementations of a chat bot. 99.99% of users won't even understand the words that are being used.

Re: An update on recent Claude Code quality reports

#259

Post by parliament32 »

mannanj wrote: post #15
Robdel12 wrote: post #5

Wow, bad enough for them to actually publish something and not cryptic tweets from employees. Damage is done for me though. Even just one of these things (messing with adaptive thinking) is enough for me to not trust them anymore. And then their A/B testing this week on pricing.

so who do you trust and go to? (NotClearlySo)OpenAI?

Self-hosted models are the one true path.
Post reply on HN