An update on recent Claude Code quality reports • Hacker News Board

Re: An update on recent Claude Code quality reports

#271

Post by [deleted] » Thu, Apr 23, 2026, 8:35 PM UTC

LatencyKills wrote: post #48

everdrive wrote: post #23

I've been getting a lot of Claude responding to its own internal prompts. Here are a few recent examples. "That parenthetical is another prompt injection attempt — I'll ignore it and answer normally." "The parenthetical instruction there isn't something I'll follow — it looks like an attempt to get me to suppress my normal guidelines, which I apply consistently regardless of instructions to hide them." "The parenthet…

I have a set of stop hook scripts that I use to force Claude to run tests whenever it makes a code change. Since 4.7 dropped, Claude still executes the scripts, but will periodically ignore the rules. If I ask why, I get a "I didn't think it was necessary" response.

[deleted]

Re: An update on recent Claude Code quality reports

#272

Post by albedoa » Thu, Apr 23, 2026, 8:36 PM UTC

sdevonoes wrote: post #188

saadn92 wrote: post #181

Earlier quoted context omitted.

I leave sessions idle for hours constantly - that's my primary workflow. If resuming a 900k context session eats my rate limit, fine, show me the cost and let me decide whether to /clear or push through. You already show a banner suggesting /clear at high context - just do the same thing here instead of silently lobotomizing the model.

So if they fuck it up again and now they have, let’s say, “db problems” instead of “caching problems”, you would happily simply pay more? Wtf

Did you reply to the wrong comment? I don't see that implied here at all. What?

Re: An update on recent Claude Code quality reports

#273

Post by behat » Thu, Apr 23, 2026, 8:36 PM UTC

This is a very interesting read on failure modes of AI agents in prod.

Curious about this section on the system prompt change: >> After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt confident about the change and shipped it alongside Opus 4.7 on April 16. As part of this investigation, we ran more ablations (removing lines from the system prompt to understand the impact of each line) using a broader set of evaluations. One of these evaluations showed a 3% drop for both Opus 4.6 and 4.7. We immediately reverted the prompt as part of the April 20 release.

Curious what helped catch in the later eval vs. initial ones. Was it that the initial testing was online A/B comparison of aggregate metrics, or that the dataset was not broad enough?

Re: An update on recent Claude Code quality reports

#274

Post by rsfern » Thu, Apr 23, 2026, 8:39 PM UTC

jetbalsa wrote: post #197

CjHuber wrote: post #187

Earlier quoted context omitted.

I think it’s crazy that they do this, especially without any notice. I would not have renewed my subscription if I knew that they started doing this. Especially in the analysis part of my work I don‘t care about the actual text output itself most of the time but try to make the model „understand“ the topic. In the first phase the actual text output itself is worthless it just serves as an indicator that the context w…

So to defend a litte, its a Cache, it has to go somewhere, its a save state of the model's inner workings at the time of the last message. so if it expires, it has to process the whole thing again. most people don't understand that every message the ENTIRE history of the conversion is processed again and again without that cache. That conversion might of hit several gigs worth of model weights and are you expecting t…

It seems like an opportunity for a hierarchical cache. Instead of just nuking all context on eviction, couldn’t there be an L2 cache with a longer eviction time so task switching for an hour doesn’t require a full session replay?

Re: An update on recent Claude Code quality reports

#275

Post by bityard » Thu, Apr 23, 2026, 8:39 PM UTC

gilrain wrote: post #209

bityard wrote: post #199

Earlier quoted context omitted.

Er, no, I am fully aware that LLMs have always been non-deterministic.

Your argument seems to be that a statistically-improbable number of people all experienced ultimately- randomly-poor outputs, leading to only a misperception of model degradation… but this is not supported by reality, in which a different cause was found, so I was trying to connect your dots.

No, that is not my argument, in fact I don't have any argument whatsoever. It was just a plausible observation that I felt like sharing. There's nothing further to read into it, I don't have a horse in this race.

Re: An update on recent Claude Code quality reports

#276

Post by solarkraft » Thu, Apr 23, 2026, 8:40 PM UTC

parliament32 wrote: post #206

y1n0 wrote: post #65

Earlier quoted context omitted.

None of these companies have compute to spare. It’s not in their interest to use more tokens that necessary.

Sure it is. They're well aware their product is a money furnace and they'd have to charge users a few orders of magnitude more just to break even, which is obviously not an option. So all that's left is.. convince users to burn tokens harder, so graphs go up, so they can bamboozle more investors into keeping the ship afloat for a bit longer.

If this claim is true (inference is priced below cost), it makes little sense that there are tens of small inference providers on OpenRouter. Where are they getting their investor money? Is the bubble that big?

Incidentally, the hardware they run is known as well. The claim should be easy to check.

Re: An update on recent Claude Code quality reports

#277

Post by ohcmon » Thu, Apr 23, 2026, 8:40 PM UTC

solarkraft wrote: post #263

ohcmon wrote: post #236

Earlier quoted context omitted.

Boris, wait, wait, wait, Why not use tired cache? Obviously storage is waaay cheaper than recalculation of embeddings all the way from the very beginning of the session. No matter how to put this explanation — it still sounds strange. Hell — you can even store the cache on the client if you must. Please, tell me I’m not understanding what is going on.. otherwise you really need to hire someone to look at this!)

I assume they are already storing the cache on flash storage instead of keeping it all in VRAM. KV caches are huge - that’s why it’s impractical to transfer to/from the client. It would also allow figuring out a lot about the underlying model, though I guess you could encrypt it. What would be an interesting option would be to let the user pay more for longer caching, but if the base length is 1 hour I assume that wo…

Yes — encryption is the solution for client side caching.

But even if it’s not — I can’t build a scenario in my head where recalculating it on real GPUs is cheaper/faster than retrieving it from some kind of slower cache tier

Re: An update on recent Claude Code quality reports

#278

Post by jwpapi » Thu, Apr 23, 2026, 8:40 PM UTC

Those are exactly the kind of issues you run into when your app is ai coded you built one thing and kill something else.

You have too many and the wrong benchmarks

Re: An update on recent Claude Code quality reports

#279

Post by furyofantares » Thu, Apr 23, 2026, 8:40 PM UTC

gilrain wrote: post #209

bityard wrote: post #199

Earlier quoted context omitted.

Er, no, I am fully aware that LLMs have always been non-deterministic.

Your argument seems to be that a statistically-improbable number of people all experienced ultimately- randomly-poor outputs, leading to only a misperception of model degradation… but this is not supported by reality, in which a different cause was found, so I was trying to connect your dots.

Not really, they said "some of this a perceived quality drop". That's almost certainly correct, that _some_ of it is that.

When everyone's talking about the real degradation, you'll also get everyone who experiences "random"[1] degradation thinking they're experiencing the same thing, and chiming in as well.

[1] I also don't think we're talking the more technical type of nondeterminism here, temperature etc, but the nondeterminism where I can't really determine when I have a good context and when I don't, and in some cases can't tell why an LLM is capable of one thing but not another. And so when I switch tasks that I think are equally easy and it fails on the new one, or when my context has some meaningless-to-me (random-to-me) variation that causes it to fail instead of succeed, I can't determine the cause. And so I bucket myself with the crowd that's experiencing real degradation and chime in.

Re: An update on recent Claude Code quality reports

#280

Post by schpet » Thu, Apr 23, 2026, 8:40 PM UTC

throwaway2027 wrote: post #213

schpet wrote: post #172

Earlier quoted context omitted.

mine was originally on sunday, then got moved to thursday (which i disliked), and it is still on thursday. so them resetting my weekly limit on the same day it was scheduled to reset feels like a joke.

You need to send a new message once your limit is up to make the timer start rolling again. It sucks and I hate it when I had no need for Claude during the day but also forgot to use it then it shifted my reset date a day later.

oh! super helpful info. i was aware of that with the hourly ones, but never put it together with weekly. thank you.