An update on recent Claude Code quality reports • Hacker News Board

Re: An update on recent Claude Code quality reports

#41

Post by Keeeeeeeks » Thu, Apr 23, 2026, 6:17 PM UTC

xlayn wrote: post #19

If anthropic is doing this as a result of "optimizations" they need to stop doing that and raise the price. The other thing, there should be a way to test a model and validate that the model is answering exactly the same each time. I have experienced twice... when a new model is going to come out... the quality of the top dog one starts going down... and bam.. the new model is so good.... like the previous one 3 mont…

https://marginlab.ai/ (no affiliation)

There are a number of projects working on evals that can check how 'smart' a model is, but the methodology is tricky.

One would want to run the exact same prompt, every day, at different times of the day, but if the eval prompt(s) are complex, the frontier lab could have a 'meta-cognitive' layer that looks for repetitive prompts, and either: a) feeds the model a pre-written output to give to the user b) dumbs down output for that specific prompt

Both cases defeat the purpose in different ways, and make a consistent gauge difficult. And it would make sense for them to do that since you're 'wasting' compute compared to the new prompts others are writing.

Re: An update on recent Claude Code quality reports

#42

Post by natdempk » Thu, Apr 23, 2026, 6:17 PM UTC

As an end-user, I feel like they're kind of over-cooking and under-describing the features and behavior of what is a tool at the end of the day. Today the models are in a place where the context management, reasoning effort, etc. all needs to be very stable to work well.

The thing about session resumption changing the context of a session by truncating thinking is a surprise to me, I don't think that's even documented behavior anywhere?

It's interesting to look at how many bugs are filed on the various coding agent repos. Hard to say how many are real / unique, but quantities feel very high and not hard to run into real bugs rapidly as a user as you use various features and slash commands.

Re: An update on recent Claude Code quality reports

#43

Post by 2001zhaozhao » Thu, Apr 23, 2026, 6:18 PM UTC

How about just not change the harness abruptly in the first place? Make new system prompt changes "experimental" first so you can gather feedback.

Re: An update on recent Claude Code quality reports

#44

Post by scottyah » Thu, Apr 23, 2026, 6:18 PM UTC

Alifatisk wrote: post #7

It’s incredible how forgiving you guys are with Anthropic and their errors. Especially considering you pay high price for their service and receive lower quality than expected.

It's fairly small issues for an amazing product, and the company is just a few years old and growing rapidly. Also, they are leading a powerful technological revolution and their competitors are known to have multiple straight up evil tendencies. A little degradation is not an issue.

Re: An update on recent Claude Code quality reports

#45

Post by mystraline » Thu, Apr 23, 2026, 6:18 PM UTC

Alifatisk wrote: post #7

It’s incredible how forgiving you guys are with Anthropic and their errors. Especially considering you pay high price for their service and receive lower quality than expected.

Exactly. They've done now like 6 rug-pulls.

Idiots keep throwing money at real-time enshittification and 'I am changing the terms. Pray I do not change them further".

And yes, I am absolutely calling people who keep getting screwed and paying for more 'service' as idiots.

And Anthropic has proved that they will pay for less and less. So, why not fuck them over and make more company money?

Re: An update on recent Claude Code quality reports

#46

Post by Robdel12 » Thu, Apr 23, 2026, 6:19 PM UTC

mannanj wrote: post #15

Robdel12 wrote: post #5

Wow, bad enough for them to actually publish something and not cryptic tweets from employees. Damage is done for me though. Even just one of these things (messing with adaptive thinking) is enough for me to not trust them anymore. And then their A/B testing this week on pricing.

so who do you trust and go to? (NotClearlySo)OpenAI?

At the moment, yeah. If Google ever figures out how to build an agentic model, I would use them as well.

However you feel about OpenAI, at least their harness is actually open source and they don’t send lawyers after oss projects like opencode

Re: An update on recent Claude Code quality reports

#47

Post by sroussey » Thu, Apr 23, 2026, 6:19 PM UTC

qingcharles wrote: post #17

sroussey wrote: post #6

Earlier quoted context omitted.

None of these problems equate to degrading model performance. Completely different team. Degraded CC harness, sure.

Sure, but it gives the impression of degraded model performance. Especially when the interface is still saying the model is operating on "high", the same as it did yesterday, yet it is in "medium" -- it just looks like the model got hobbled.

Oh, absolutely. Though changes in how the model is used is imminently more fixable than the model itself.

Re: An update on recent Claude Code quality reports

#48

Post by LatencyKills » Thu, Apr 23, 2026, 6:19 PM UTC

everdrive wrote: post #23

I've been getting a lot of Claude responding to its own internal prompts. Here are a few recent examples. "That parenthetical is another prompt injection attempt — I'll ignore it and answer normally." "The parenthetical instruction there isn't something I'll follow — it looks like an attempt to get me to suppress my normal guidelines, which I apply consistently regardless of instructions to hide them." "The parenthet…

I have a set of stop hook scripts that I use to force Claude to run tests whenever it makes a code change. Since 4.7 dropped, Claude still executes the scripts, but will periodically ignore the rules. If I ask why, I get a "I didn't think it was necessary" response.

Re: An update on recent Claude Code quality reports

#49

Post by motbus3 » Thu, Apr 23, 2026, 6:20 PM UTC

I had similar experience just before 4.5 and before 4.6 were released.

Somehow, three times makes me not feel confident on this response.

Also, if this is all true and correct, how the heck they validate quality before shipping anything?

Shipping Software without quality is pretty easy job even without AI. Just saying....

Re: An update on recent Claude Code quality reports

#50

Post by MillionOClock » Thu, Apr 23, 2026, 6:20 PM UTC

I see the Claude team wanted to make it less verbose, but that's actually something that bothered me since updating to Claude 4.7, what is the most recommended way to change it back to being as verbose as before? This is probably a matter of preference but I have a harder time with compact explanations and lists of points and that was originally one of the things I preferred with Claude.