An update on recent Claude Code quality reports • Hacker News Board

Re: An update on recent Claude Code quality reports

#721

Post by baxtr » Fri, Apr 24, 2026, 8:59 PM UTC

musebox35 wrote: post #711

dgreensp wrote: post #658

This reveals a staggering level of incompetence, if that’s really all it is, and lack of transparency. They don’t have ANY product-level quality tests that picked this up? Many users did their own tests and published them. It’s not hard. And these users’ complaints were initially dismissed. I don’t think the high vs medium change is really on par with the others. That’s a setting you change in the UI, and depending o…

They say that they did test but the coverage was not enough to pick it up, at least for the prompt change: “ After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt confident about the change and shipped it alongside Opus 4.7 on April 16. As part of this investigation, we ran more ablations (removing lines from the system prompt to understand the impact of each line) usin…

Naively, one could assume that with AI it should be possible to create a long and broad list of test cases…

Re: An update on recent Claude Code quality reports

#722

Post by bshanks » Fri, Apr 24, 2026, 9:14 PM UTC

bcherny wrote: post #124

6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

The main issue here is not UX, but rather that you did something which degraded quality without transparency. You should have documented this and also highlighted the change in an announcement. There should never be an undocumented change that reduces quality. There should never be something the user can do (or fail to do) that reduces quality without that being documented. To regain trust, Anthropic should make an announcement committing to documenting/announcing any future intentional quality-reducing changes.

In addition, the following is less important, but as other commenters have stated: walking away from a conversation and coming back to it more than an hour later is very common and it would be nice if there were a way for the user to opt to retain maximum quality (e.g. no dropped thinking) in this case. In the longer term, it would be nice if there were a way for the user to wait a few minutes for a stale session to resume, in exchange for not having a large amount of quota drained (ie have a 'slow mode' invoked upon session resumption that consumes less quota).

Re: An update on recent Claude Code quality reports

#723

Post by tedivm » Fri, Apr 24, 2026, 9:27 PM UTC

libraryofbabel wrote: post #691

tedivm wrote: post #680

Earlier quoted context omitted.

That cost that you're talking about doesn't change based on how long the session is idle. No matter what happens they're storing that state and bring it back at some point, the only difference is how long it's stored out of GPU between requests.

Are you sure about that? They charge $6.25 / MTok for 5m TTL cache writes and $10 / MTok for 1hr TTL writes for Opus. Unless you believe Anthropic is dramatically inflating the price of the 1hr TTL, that implies that there is some meaningful cost for longer caches and the numbers are such that it's not just the cost of SSD storage or something. Obviously the details are secret but if I was to guess, I'd say the 5m ca…

Storing on GPU would be the absolute dumbest thing they could do. Locking up the GPU memory for a full hour while waiting for someone else to make a request would result in essentially no GPU memory being available pretty rapidly. This type of caching is available from the cloud providers as well, and it isn't tied to a single session or GPU.

Re: An update on recent Claude Code quality reports

#724

Post by xpe » Fri, Apr 24, 2026, 10:26 PM UTC

willis936 wrote: post #625

xpe wrote: post #443

Earlier quoted context omitted.

> All of this points to their priorities not being aligned with their users’. Framing this as "aligned" or "not aligned" ignores the interesting reality in the middle. It is banal to say an organization isn't perfectly aligned with its customers. I'm not disagreeing with the commenter's frustration. But I think it can help to try something out: take say the top three companies whose product you interact with on a reg…

So you're arguing they're just plain incompetent? Not sure that's going to win the trust of customers either.

> So you're arguing they're just plain incompetent? Not sure that's going to win the trust of customers either.

This is not a charitable interpretation of what I wrote. Please take a minute and rethink and rephrase. Here are two important guidelines, hopefully familiar to someone who has had an account since 2019:

> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.

> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

Re: An update on recent Claude Code quality reports

#725

Post by tetromino_ » Fri, Apr 24, 2026, 10:33 PM UTC

pixl97 wrote: post #703

sfink wrote: post #674

Earlier quoted context omitted.

> Why does it take the company that is probably the best at agentic coding more than a month to find and solve such large regressions, even with customers complaining about them? My unfounded suspicion: because this is the tradeoff we're all facing and for the most part refusing to accept when transitioning over to LLM-driven coding. This is exactly how we're being trained to work by the strengths and limitations of…

> you might have a `--cache-dir=DIR` option that doesn't actually do anything Working in enterprise software it's surprising how long an option that doesn't actually do anything can be missed. And that was before AI and having thousands of customers use it. This same problem happens with documentation all the time. You end up with paragraphs or examples that simply don't reflect what the product actually does.

Where I work, options that don't do anything are seen as good engineering practice. You see, you can't break your user's scripts. Your CLI arguments are part of your stable API. If your tool used to have a cache_dir CLI option, and now no longer needs it, you still have to keep accepting cache_dir and treat it as a no-op until you are confident your users have migrated away from it.

Re: An update on recent Claude Code quality reports

#726

Post by xpe » Fri, Apr 24, 2026, 10:34 PM UTC

daveoc64 wrote: post #632

xpe wrote: post #443

Earlier quoted context omitted.

> All of this points to their priorities not being aligned with their users’. Framing this as "aligned" or "not aligned" ignores the interesting reality in the middle. It is banal to say an organization isn't perfectly aligned with its customers. I'm not disagreeing with the commenter's frustration. But I think it can help to try something out: take say the top three companies whose product you interact with on a reg…

>My overall feel is that people underestimate the complexity of the systems at Anthropic and the chaos of the growth. Making changes like reducing the usage window at peak times ( https://x.com/trq212/status/2037254607001559305 ) without announcing it (until after the backlash) is the sort of thing that's making people lose trust in Anthropic. They completely ignored support tickets and GitHub issues about that for 3…

Thanks for the example -- you are one of the first people to quote a source, so I appreciate it. This makes constructive discussion much easier. You quoted this:

    > To manage growing demand for Claude we're adjusting our
    > 5 hour session limits for free/Pro/Max subs during peak
    > hours. Your weekly limits remain unchanged.
    >
    > During weekdays between 5am–11am PT / 1pm–7pm GMT, you'll
    > move through your 5-hour session limits faster than before.

And yeah, no disagreement from me: many users are not going to like this. Narrowly speaking, I don't want any chance that reduces what I get for what I pay for. I also care about overall reliability, so if some users on the right tail of the usage distribution find themselves losing out, my take is "Yeah, they are disappointed, but this is rational decision for any company with this kind of subscription model."

Broken expectations are highly dependent on perception. People get used to having some particular level. When that changes and they notice, and being humans a strong default is to reach for something to blame. Then we rationalize. That last two parts are unhelpful, and I push back on them frequently.

Re: An update on recent Claude Code quality reports

#727

Post by giwook » Fri, Apr 24, 2026, 10:43 PM UTC

jimkleiber wrote: post #433

giwook wrote: post #333

Earlier quoted context omitted.

Another way to think about it might be that caching is part of Anthropic's strategy to reduce costs for its users, but they are now trying to be more mindful of their costs (probably partly due to significant recent user growth as well as plans to IPO which demand fiscal prudence). Perhaps if we were willing to pay more for our subscriptions Anthropic would be able to have longer cache windows but IDK one hour seems…

I might be willing to pay more, maybe a lot more, for a higher subscription than claude max 20x, but the only thing higher is pay per token and i really dont like products that make me have to be that minutely aware of my usage, especially when it has unpredictability to it. I think there's a reason most telecoms went away from per minute or especially per MB charging. Even per GB, as they often now offer X GB, and i…

For sure, I agree with that sentiment. It's interesting to consider the psychological component of that, like how "free shipping" is not really free, it's oftentimes just packaged into the price of the product but somehow it feels like we're getting a better deal.

I would not be surprised to see Anthropic, OpenAI etc head in the direction you mention as they mature and all of these datacenters currently undergoing construction come online in the next few years and drive down costs.

Re: An update on recent Claude Code quality reports

#728

Post by xpe » Fri, Apr 24, 2026, 10:58 PM UTC

daveoc64 wrote: post #632

xpe wrote: post #443

Earlier quoted context omitted.

> All of this points to their priorities not being aligned with their users’. Framing this as "aligned" or "not aligned" ignores the interesting reality in the middle. It is banal to say an organization isn't perfectly aligned with its customers. I'm not disagreeing with the commenter's frustration. But I think it can help to try something out: take say the top three companies whose product you interact with on a reg…

>My overall feel is that people underestimate the complexity of the systems at Anthropic and the chaos of the growth. Making changes like reducing the usage window at peak times ( https://x.com/trq212/status/2037254607001559305 ) without announcing it (until after the backlash) is the sort of thing that's making people lose trust in Anthropic. They completely ignored support tickets and GitHub issues about that for 3…

> You shouldn't have to rely on finding an individual employee's posts on Reddit or X for policy announcements.

I agree with this as a principle. Which raises this question: is it true? Are you certain these messages don't show up in (a) Claude Code and (b) Claude on the Web?

I've seen these kinds of messages pop up. I haven't taken inventory of how often they do. As a guess, maybe I see notifications like this several times a month. If any important ones are missing, that is a mistake.

Anyhow, this is the kind of discussion that I want people to have. I appreciate the detail.

> A company with their resources could easily do better.

Yes, they could. But easily? I'm not so sure.

Also ask yourself: what function does saying e.g. "they could have done better" serve? What does it help accomplish? I'm asking. I think it often serves as a sort of self-reinforcing thing to say that doesn't really invite more thinking.

Ask yourself: If "doing better" was easy, why didn't it happen? Maybe it isn't quite as easy as you think? Maybe you've baked in a lot of assumptions. Easy for who? Easy why? Try the questions I asked, above. They are not rhetorical. Here they are again, rephrased a bit

    > take the top three companies whose product you 
    > interact with on a regular basis. Take stock of
    > (1) how fast the technology is moving;
    > (2) how often things break from your POV;
    > (3) how soon the company acknowledges it;
    > (4) how long it takes for a fix.
    >
    > Then ask "if a friend of mine (competent, hard working)
    > worked there, how would I be thinking about the situation?"

There is a reason why I recommend asking these questions. Forcing yourself to write down your reference class is ... to me, table stakes, but well, lots of people just leave it floating and then ask other people to magically reconstruct it. Envisioning a friend working there shifts your viewpoint and can shake lose many common biases.

Re: An update on recent Claude Code quality reports

#729

Post by kziad » Fri, Apr 24, 2026, 11:19 PM UTC

LatencyKills wrote: post #48

everdrive wrote: post #23

I've been getting a lot of Claude responding to its own internal prompts. Here are a few recent examples. "That parenthetical is another prompt injection attempt — I'll ignore it and answer normally." "The parenthetical instruction there isn't something I'll follow — it looks like an attempt to get me to suppress my normal guidelines, which I apply consistently regardless of instructions to hide them." "The parenthet…

I have a set of stop hook scripts that I use to force Claude to run tests whenever it makes a code change. Since 4.7 dropped, Claude still executes the scripts, but will periodically ignore the rules. If I ask why, I get a "I didn't think it was necessary" response.

[dead]

Re: An update on recent Claude Code quality reports

#730

Post by charcircuit » Sat, Apr 25, 2026, 12:12 AM UTC

pixl97 wrote: post #709

charcircuit wrote: post #534

Earlier quoted context omitted.

>and doing that will cause a huge one-time hit against your token limit if the session has grown large. Anthropic already profited from generating those tokens. They can afford subsidize reloading context.

No they can't, that's what you don't seem to get. Reloading those tokens takes around the same effort as processing them in the first place. It's ok to be ignorant of how the infrastructure for LLMs work, just don't be proud of it.

They literally can. They could make the API free to use if they wanted. There is no law that states that costs have to equal the cost it takes to process the request.