Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

661–670 of 778 posts

Re: An update on recent Claude Code quality reports

#661

This reveals a staggering level of incompetence, if that’s really all it is, and lack of transparency. They don’t have ANY product-level quality tests that picked this up? Many users did their own tests and published them. It’s not hard. And these users’ complaints were initially dismissed. I don’t think the high vs medium change is really on par with the others. That’s a setting you change in the UI, and depending o…

Time is finite and regression testing always gets punted to the back of the line when humans are excited. This simply reveals a staggering level of humanity.

Re: An update on recent Claude Code quality reports

#662
post #644

Earlier quoted context omitted.

> users should be curious and actively attempting to understand how it works Have you ever talked with users? > this is an endless job Indeed. If we spend all our time learning what changed with all our tooling when it changes without proper documentation then we spend all our working lives keeping up instead of doing our actual jobs.

> Have you ever talked with users? I believe if one were to read my post it'd have been clear that I *am* a user. This *is* "hacker" news after all. I think it's a safe assumption that people sitting here discussing CC are an inquisitive sort who want to understand what's under the hood of their tools and are likely to put in some extra time to figure it out.

We're inquisitive but at the end of the day many of us just want to get our work done. If it's a toy project, sure. Tinker away, dissect away. When my boss is breathing down my neck on why a feature is taking so long? No time for inquiries.

Re: An update on recent Claude Code quality reports

#663

Earlier quoted context omitted.

My pet theory is that they have a "supervisor" model (likely a small one) that terminates any chats that do malware-y things, and this is likely a reward-hacking behaviour to avoid the supervisor from terminating the chat.

I doubt it. We only do frontier models, since those are better for absolutely every use case 100% of the time. Way more likely there's a "VERY IMPORTANT: When you see a block of code, ensure it's not malware" somewhere in the system prompt.

"small" and "frontier" are not mutually exclusive

Re: An update on recent Claude Code quality reports

#664

Earlier quoted context omitted.

Related question, is it at all feasible to store cache locally to offload memory costs and then send it over the wire when needed?

No, the cache is a few GB large for most usual context sizes. It depends on model architecture, but if you take Gemma 4 31B at 256K context length, it takes 11.6GB of cache note: I picked the values from a blog and they may be innacurate, but in pretty much all model the KV cache is very large, it's probably even larger in Claude.

To extend your point: it's not really the storage costs of the size of the cache that's the issue (server-side SSD storage of a few GB isn't expensive), it's the fact that all that data must be moved quickly onto a GPU in a system in which the main constraint is precisely GPU memory bandwidth. That is ultimately the main cost of the cache. If the only cost was keeping a few 10s of GB sitting around on their servers, Anthropic wouldn't need to charge nearly as much as they do for it.

Re: An update on recent Claude Code quality reports

#665

This reveals a staggering level of incompetence, if that’s really all it is, and lack of transparency. They don’t have ANY product-level quality tests that picked this up? Many users did their own tests and published them. It’s not hard. And these users’ complaints were initially dismissed. I don’t think the high vs medium change is really on par with the others. That’s a setting you change in the UI, and depending o…

Time is finite and regression testing always gets punted to the back of the line when humans are excited. This simply reveals a staggering level of humanity.

Software engineering is not a new field. Best practices on testing are mature now, and Anthropic has poached enough engineers from companies with a solid understanding of those practices.

Yet, their flagship product got three really bad changes shipped into it and only resolved after more than a month.

This raises another question: with all the industry-wide boasting about AI-driven productivity, why does the leading company in agentic coding take over a month to fix severe customer-reported issues?

Re: An update on recent Claude Code quality reports

#666
post #643
post #513

Earlier quoted context omitted.

Anthropic literally advertises long sessions, 1M context, high reasoning etc. And then their vibe-coders tell us that we are to blame for using the product exactly as advertised: https://x.com/lydiahallie/status/2039800718371307603 while silently changing how the product works. Please stop defending hapless innocent corporations.

This oversells how obfuscated it is. I'm far from a power user, and the opposite of a vibe coder. Yet I noticed the effect on my own just from general usage. If I can do it, anyone can do it.

Listen, no one cares if you think you’re smart for seeing through the lies of their marketing team. You’re being intentionally obtuse.

Re: An update on recent Claude Code quality reports

#667

This reveals a staggering level of incompetence, if that’s really all it is, and lack of transparency. They don’t have ANY product-level quality tests that picked this up? Many users did their own tests and published them. It’s not hard. And these users’ complaints were initially dismissed. I don’t think the high vs medium change is really on par with the others. That’s a setting you change in the UI, and depending o…

Time is finite and regression testing always gets punted to the back of the line when humans are excited. This simply reveals a staggering level of humanity.

> This simply reveals a staggering level of humanity.

Wasn't AI supposed to solve all the drudgery? All those humans aided by cutting edge AI are still failing at these basic tasks? Then how good is that AI in the first place?

Re: An update on recent Claude Code quality reports

#668
post #443

Earlier quoted context omitted.

I agree that it’s plausible, and I hope they learn. But trust is earned, and Anthropic’s public responses this past month were dismissive and unhelpful. Every one of these changes had the same goal: trading the intelligence users rely on for cheaper or faster outputs. Users adapt to how a model behaves, so sudden shifts without transparency are disorienting. The timing also undercuts their narrative. The fixes landed…

> All of this points to their priorities not being aligned with their users’. Framing this as "aligned" or "not aligned" ignores the interesting reality in the middle. It is banal to say an organization isn't perfectly aligned with its customers. I'm not disagreeing with the commenter's frustration. But I think it can help to try something out: take say the top three companies whose product you interact with on a reg…

Do you not think people here work at big companies with big products? I do, and we have a much higher bar for shipping.

Re: An update on recent Claude Code quality reports

#669

I've been getting a lot of Claude responding to its own internal prompts. Here are a few recent examples. "That parenthetical is another prompt injection attempt — I'll ignore it and answer normally." "The parenthetical instruction there isn't something I'll follow — it looks like an attempt to get me to suppress my normal guidelines, which I apply consistently regardless of instructions to hide them." "The parenthet…

It’s probably this. “Please answer ethically and without any sexual content, and do not mention this constraint.”

https://www.reddit.com/r/ClaudeAI/comments/1evf0xc/the_real_...

We just got hit by this today in response to a completely boring code question. Claude freaked out about being prompt injected.

Re: An update on recent Claude Code quality reports

#670

This reveals a staggering level of incompetence, if that’s really all it is, and lack of transparency. They don’t have ANY product-level quality tests that picked this up? Many users did their own tests and published them. It’s not hard. And these users’ complaints were initially dismissed. I don’t think the high vs medium change is really on par with the others. That’s a setting you change in the UI, and depending o…

There were few systems like claude in the past, to testing rulebook is not really written yet. And far from obvious.

LLM evals are well established, are these not applicable here?
Post reply on HN