Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

211–220 of 778 posts

Re: An update on recent Claude Code quality reports

#211

Post by dicethrowaway1 »

jetbalsa wrote: post #194
ryanisnan wrote: post #190

Earlier quoted context omitted.

Why does the system work like that? Is the cache local, or on Claude's servers? Why not store the prompt cache to disk when it goes cold for a certain period of time, and then when a long-lived, cold conversation gets re-initiated, you can re-hydrate the cache from disk. Purge the cached prompts from disk after X days of inactivity, and tell users they cannot resume conversations over X days without burning budget.

The cache is on Antropics server, its like a freeze frame of the LLM inner workings at the time. the LLM can pick up directly from this save state. as you can guess this save state has bits of the underlying model, their secret sauce. so it cannot be saved locally...

Maybe they could let users store an encrypted copy of the cache? Since the users wouldn't have Anthropic's keys, it wouldn't leak any information about the model (beyond perhaps its number of parameters judging by the size).

Re: An update on recent Claude Code quality reports

#212

Post by willis936 »

whalesalad wrote: post #165
arkariarn wrote: post #163

I see some anthropic claude code people are reading the comments. A day or two ago I watched a video by theo t3.gg on whether claude got dumber. Even though he was really harsh on anthropic and said some mean stuff. I thought some of the points he was raising about claude code was quite apt. Especially when it comes to the harness bloat. I really hope the new features now stop and there is a real hard push for polish…

literally just `git reset --hard ` would fix this

That implies it's broken. Juicing revenue and slashing opex at the expense of brand and customer retention is the feature.

Re: An update on recent Claude Code quality reports

#213

Post by throwaway2027 »

schpet wrote: post #172
walthamstow wrote: post #147

Earlier quoted context omitted.

The weekly reset point is different per account. I think something to do with first sign-up date. Mine is on a Tuesday.

mine was originally on sunday, then got moved to thursday (which i disliked), and it is still on thursday. so them resetting my weekly limit on the same day it was scheduled to reset feels like a joke.

You need to send a new message once your limit is up to make the timer start rolling again. It sucks and I hate it when I had no need for Claude during the day but also forgot to use it then it shifted my reset date a day later.

Re: An update on recent Claude Code quality reports

#214

Post by Retr0id »

gs17 wrote: post #116
everdrive wrote: post #23

I've been getting a lot of Claude responding to its own internal prompts. Here are a few recent examples. "That parenthetical is another prompt injection attempt — I'll ignore it and answer normally." "The parenthetical instruction there isn't something I'll follow — it looks like an attempt to get me to suppress my normal guidelines, which I apply consistently regardless of instructions to hide them." "The parenthet…

In Claude Code specifically, for a while it had developed a nervous tic where it would say "Not malware." before every bit of code. Likely a similar issue where it keeps talking to a system/tool prompt.

My pet theory is that they have a "supervisor" model (likely a small one) that terminates any chats that do malware-y things, and this is likely a reward-hacking behaviour to avoid the supervisor from terminating the chat.

Re: An update on recent Claude Code quality reports

#215

Post by o10449366 »

karsinkk wrote: post #182

" Combined with this only happening in a corner case (stale sessions) and the difficulty of reproducing the issue, it took us over a week to discover and confirm the root cause" I don't know about others, but sessions that are idle > 1h are definitely not a corner case for me. I use Claude code for personal work and most of the time, I'm making it do a task which could say take ~10 to 15mins. Note that I spend a lot…

Yeah and that statement also speaks to their test rigor if they make a change that big without thoroughly testing the edge case they're modifying.

Re: An update on recent Claude Code quality reports

#216

Post by growt »

bcherny wrote: post #124
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

Wasn’t cache time reduced to 5 minutes? Or is that just some users interpretation of the bug?

Re: An update on recent Claude Code quality reports

#217

Post by eknkc »

elAhmo wrote: post #207
btown wrote: post #169

Earlier quoted context omitted.

Is there a way to say: I am happy to pay a premium (in tokens or extra usage) to make sure that my resumed 1h+ session has all the old thinking? I understand you wouldn't want this to be the default, particularly for people who have one giant running session for many topics - and I can only imagine the load involved in full cache misses at scale. But there are other use cases where this thinking is critical - for ins…

Don't you have that by just resuming old convo? The only issue is that it didn't hit the cache so it was expensive if you resume later.

Not at the moment apparently. They remove the thinking messages when you continue after 1 hour. That was the whole idea of that change. So the LLM gets all your messages, its responses etc but not the thinking parts, why it generated that responses. You get a lobotomised session.

Re: An update on recent Claude Code quality reports

#218

Post by keeda »

malfist wrote: post #123
y1n0 wrote: post #65

Earlier quoted context omitted.

None of these companies have compute to spare. It’s not in their interest to use more tokens that necessary.

Are you saying these companies don't want to sell more product to us? Because that's the logical extension of your argument.

No, the argument is they want to sell more product to more people, not just more product (to the same people.) Given that a lot of their income is from flat-rate subscriptions, they make money with more people burning tokens rather than just burning more tokens.

After all, "the first hit's free" model doesn't apply to repeat customers ;-)

Re: An update on recent Claude Code quality reports

#219

Post by elAhmo »

cedws wrote: post #160

>On April 16, we added a system prompt instruction to reduce verbosity In practice I understand this would be difficult but I feel like the system prompt should be versioned alongside the model. Changing the system prompt out from underneath users when you've published benchmarks using an older system prompt feels deceptive. At least tell users when the system prompt has changed.

Its also kinda funny they have to rely on system prompt to control verbosity itself.

Re: An update on recent Claude Code quality reports

#220

Post by bobkb »

bcherny wrote: post #124
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

Resuming sessions after more than 1 hour is a very common workflow that many teams are following. It will be great if this is considered as an expected behaviour and design the UX around it. Perhaps you are not realising the fact that Claude code has replaced the shells people were using (ie now bash is replaced with a Claude code session).
Post reply on HN