Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

731–740 of 778 posts

Re: An update on recent Claude Code quality reports

#731

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

Just curious, is there a consolidated list of all these "education" tips?

Intuitively I understand this due to how context windows work and you're looking to increase cache hits, has Anthropic tried compact/summarise on idle as a configurable option? Seems to have decent tradeoffs + education in a setting.

Re: An update on recent Claude Code quality reports

#732
post #674

Earlier quoted context omitted.

Software engineering is not a new field. Best practices on testing are mature now, and Anthropic has poached enough engineers from companies with a solid understanding of those practices. Yet, their flagship product got three really bad changes shipped into it and only resolved after more than a month. This raises another question: with all the industry-wide boasting about AI-driven productivity, why does the leading…

> Why does it take the company that is probably the best at agentic coding more than a month to find and solve such large regressions, even with customers complaining about them? My unfounded suspicion: because this is the tradeoff we're all facing and for the most part refusing to accept when transitioning over to LLM-driven coding. This is exactly how we're being trained to work by the strengths and limitations of…

  > digests user input and boils out the noise to provide a robust enough signal to guide some monitoring agent
not to sound uncharitable but this seems like the absolute worst way to run a business; your customers are basically lab rats... why should they pay for anything in this scenario?

Re: An update on recent Claude Code quality reports

#733
post #475

Earlier quoted context omitted.

> upload and restore it when the user starts their next interaction The data is the conversation (along with the thinking tokens). There is no download - you already have it. The issue is that it gets expunged from the (very expensive, very limited) GPU cache and to reload the cache you have to reprocess the whole conversation. That is doable, but as Boris notes it costs lots of tokens.

You're quite confidently wrong! :-) The kv-cache is the internal LLM state after having processed the tokens. It's big, and you do not have it locally.

> The kv-cache is the internal LLM state after having processed the tokens. It's big, and you do not have it locally.

Yes - generated from the data of the conversation.

Read what I said again. I'm explaining how they regenerate the cache by running the conversation though the LLM to reconstruct the KV cache state.

Re: An update on recent Claude Code quality reports

#734
post #724

Earlier quoted context omitted.

So you're arguing they're just plain incompetent? Not sure that's going to win the trust of customers either.

> So you're arguing they're just plain incompetent? Not sure that's going to win the trust of customers either. This is not a charitable interpretation of what I wrote. Please take a minute and rethink and rephrase. Here are two important guidelines, hopefully familiar to someone who has had an account since 2019: > Comments should get more thoughtful and substantive, not less, as a topic gets more divisive. > Please…

I didn't assume bad faith, I simply reworded your conclusions with less soft language so that others would understand your position more clearly.

You are saying what they are doing is hard. That's fine. Their stated goals are to be the responsible stewards of the technology and we agree they are failing at that goal. You would attribute that to incompetence and not malice.

Re: An update on recent Claude Code quality reports

#735
post #643

Earlier quoted context omitted.

This oversells how obfuscated it is. I'm far from a power user, and the opposite of a vibe coder. Yet I noticed the effect on my own just from general usage. If I can do it, anyone can do it.

Listen, no one cares if you think you’re smart for seeing through the lies of their marketing team. You’re being intentionally obtuse.

My point is the opposite. I don't think my observation was smart, and I'm surprised to so many people here, a venue with a lot of people who use this stuff far more than I do, think it wasn't an easy to grok thing.

Re: An update on recent Claude Code quality reports

#736
post #306

Earlier quoted context omitted.

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

> tokens written to cache all at once, which would eat up a significant % of your rate limits Construction of context is not an llm pass - it shouldn't even count towards token usage. The word 'caching' itself says don't recompute me. Since the devs on HN (& the whole world) is buying what looks like nonsense to me - what am I missing?

> Since the devs on HN (& the whole world) is buying what looks like nonsense to me - what am I missing?

Input tokens are expensive, since the whole model has to be run for each token. They're cheaper than output tokens because the model doesn't need to run the sampler, so some pipeline parallelism is possible, but on the other hand without caching the input token cost would have to be paid anew for each output token.

Prompt caching fixes that O(N^2) cost, but the cache itself is very heavyweight. It needs one entry per input token per model layer, and each entry is an O(1000)-dimensional vector. That carries a huge memory cost (linear in context length), and when cached that means the context's memory space is no longer ephemeral.

That's why a 'cache write' can carry a cost; it is the cost of both processing the input and committing the backing store for the cache duration.

Re: An update on recent Claude Code quality reports

#737
post #674

Earlier quoted context omitted.

> Why does it take the company that is probably the best at agentic coding more than a month to find and solve such large regressions, even with customers complaining about them? My unfounded suspicion: because this is the tradeoff we're all facing and for the most part refusing to accept when transitioning over to LLM-driven coding. This is exactly how we're being trained to work by the strengths and limitations of…

> digests user input and boils out the noise to provide a robust enough signal to guide some monitoring agent not to sound uncharitable but this seems like the absolute worst way to run a business; your customers are basically lab rats... why should they pay for anything in this scenario?

I just said someone's gonna build it, not that it's a good idea!

To be fair [to myself], this is scale-dependent. I work on a product with hundreds of millions of users. We're not going to be reading and pondering every bit of feedback we get. We have automation for stripping out some of the noise (eg the number of crash reports we get from bit flips due to faulty RAM is quite significant at this scale). We have lines of defense set up to screen things down -- though if you file a well-researched and documented bug, we'll pay attention. (We won't necessarily do what you want, but we'll pay attention.)

When I worked at a much smaller and earlier stage company, we begged our users for feedback. We begged potential users for feedback. We implemented some things purely to try to get someone excited enough that they would be motivated to give feedback.

Anthropic, OpenAI, Google? They have a lot of users.

Also, this automation would be in addition to the other channels by which you'd pay attention to feedback.

Also also, the ship has sailed. We're all lab rats now. We're randomly chosen to be A/B tested on. We are upgraded early as part of a staged rollout. We're region-locked. Geocoded. Tracked as part of the cohort that has bought formula or diapers recently. Maybe we live in the worst of all possible worlds?

Re: An update on recent Claude Code quality reports

#739

Earlier quoted context omitted.

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

2. could you bring back the _compact and accept plan_? even if it is not the default option.

Add this to your `settings.json`:

"showClearContextOnPlanAccept": true,

Re: An update on recent Claude Code quality reports

#740
post #724

Earlier quoted context omitted.

> So you're arguing they're just plain incompetent? Not sure that's going to win the trust of customers either. This is not a charitable interpretation of what I wrote. Please take a minute and rethink and rephrase. Here are two important guidelines, hopefully familiar to someone who has had an account since 2019: > Comments should get more thoughtful and substantive, not less, as a topic gets more divisive. > Please…

I didn't assume bad faith, I simply reworded your conclusions with less soft language so that others would understand your position more clearly. You are saying what they are doing is hard. That's fine. Their stated goals are to be the responsible stewards of the technology and we agree they are failing at that goal. You would attribute that to incompetence and not malice.

I personally try to follow Rapoport's Rules, and I since think they are consistent with the HN Guidelines, I like to mention them: [1].

I've thought on it, and I will try to start off with something we both agree on... We both agree that Anthropic made some mistakes, but this is probably a pretty uninteresting and shallow agreement. I find it unlikely that we would enumerate or characterize the mistakes similarly. I find it unlikely that we would be anywhere near the same headspace about our bigger-picture takes.

> I didn't assume bad faith

Ok, I'm glad. That one didn't concern me; if I had a do-over I would remove that one from the list. Sorry about that. These are the ones that concern me:

    > Comments should get more thoughtful and substantive,
    > not less, as a topic gets more divisive.
When I read your earlier comment (~20 words), it didn't come across as a thoughtful and substantive response to my comment (~160 words). I know length isn't a perfect measure nor the only measure, but it does matter.

    > Please respond to the strongest plausible interpretation of what
    > someone says, not a weaker one that's easier to criticize.
Are you sure you didn't choose an easier to criticize interpretation? Did you take the take to try to state to yourself what I was trying to say? Back to Rapaport's Rules ...

    > You should attempt to re-express your target’s position so
    > clearly, vividly, and fairly that your target says, “Thanks,
    > I wish I’d thought of putting it that way.”
I'm grateful when people can express what I'm going for better than the way I wrote it or said it.

> I simply reworded your conclusions with less soft language

Technically speaking, lots of things could be called "rewording", but what you did was relatively far from "simply rewording". Charitably, it is closer to "your interpretation". But my intent was lost, so "rewording" doesn't fit.

> ... so that others would understand your position more clearly.

If you want to help others understand, then it is good to make sure you understand. For that, I recommend asking questions.

> Their stated goals are to be the responsible stewards of the technology and we agree they are failing at that goal.

No, I do not agree to that phrasing. It is likely I don't agree with your intention behind it either.

> You would attribute that to incompetence and not malice.

No; even if I agreed with the premise, I think it is more likely I would still disagree. I don't even like the framing of "either malice or incompetence". These ideas don't carve reality at the joints. [2] [3] There are a lot of stereotypes about "incompetence" but I don't think they really help us understand the world. These stereotypes are more like thought-terminators than interesting generative lenses.

I'll try to bring it back to the words "malice" and "incompetence" even though I think the latter is nigh-useless as a sense-making tool. Many mistakes happen without malice or incompetence; many mistakes "just happen" because people and organizations are not designed to be perfect. They are designed to be good enough. To not make any short-term mistakes would likely require too much energy or too much rigidity, both of which would be a worse category of mistake.

Try to think counterfactually: imagine a world where Anthropic is not malicious nor incompetent and yet mistakes still happened. What would this look like?

When you think of what Anthropic did wrong, what do you see as the lead up to it? Can you really envision the chain of events that brought it about? Imagine reading the email chain or the PRs. Can you see how there may be been various "off-ramps" where history might have gone differently? But for each of those diversions, how likely would it be that they match the universe we're in?

At some point figuring out what is a "mistake" even starts to feel strange. Does it require consciousness? Most people think so. But we say organizations make mistakes, but they aren't conscious -- or are they? Who do we blame? The CEO, because the buck stops there, right? He "should have known better". But why? Wait, but the Board is responsible...?

Is there any ethical foundation here? Some standard at all or is this all just anger dressed up as an argument? If this assigning blame thing starts to feel horribly complicated or even pointless, then maybe I've made my point. :)

If nothing else, when you read what I write, I want it to make you stop, get out a sheet of paper, and try to imagine something vividly. Your imagination I think will persuade you better than I can.

[1]: https://themindcollection.com/rapoports-rules/

[2]: https://jollycontrarian.com/index.php?title=Carving_nature_a...

[3]:https://english.stackexchange.com/questions/303819/what-do-t...

Post reply on HN