Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

761–770 of 778 posts

Re: An update on recent Claude Code quality reports

#761

Post by MrOrelliOReilly »

anonyfox wrote: post #634
MrOrelliOReilly wrote: post #317

IMO this is the consequence of a relentless focus on feature development over core product refinement. I often have the impression that Anthropic would benefit from a few senior product people. Someone needs to lend them a copy of “Escaping the Build Trap.” Just because we _can_ rapidly add features now doesn’t mean we should. PS I’m not referencing a well-known book to suggest the solution is trite product group thi…

Essentially they should hire a few of the old school product guys from Apple. Best me to it, but the obsession on UX and quality from earlier Apple is exactly what they urgently need instead of tech folks trying to engineer themselves into complicated rabbit holes and shenanigans.

Exactly!

Re: An update on recent Claude Code quality reports

#762

Post by kang »

coldtea wrote: post #757
kang wrote: post #360

Earlier quoted context omitted.

You not only skipped the diligence but confused everyone repeating what I said :( that is what caching is doing. the llm inference state is being reused. (attention vectors is internal artefact in this level of abstraction, effectively at this level of abstraction its a the prompt). The part of the prompt that has already been inferred no longer needs to be a part of the input, to be replaced by the inference subset.…

>It seems you haven't done the due diligence on what part of the API is expensive - constructing a prompt shouldn't be same charge/cost as llm pass. I think you missed what the parent meant then, and the confusing way you replied seemed to imply that they're not doing inference caching (the opposite of what you wanted to mean). The parent didn't said that caching is needed to merely avoid reconstructing the prompt as…

You are right, I was wrong in my understanding there. It stemmed from my own implementation; an inference often wrote extra data such as tool call, so I was using it to preserve relevant information alongwith desired output, to be able to throw away the prompt every time. I realize inference caching is one better way (with its pros and cons).

Re: An update on recent Claude Code quality reports

#763

Post by marcyb5st »

CamperBob2 wrote: post #378
marcyb5st wrote: post #108

Earlier quoted context omitted.

Apart from Anthropic nobody knows how much the average user costs them. However the consensus is "much more than that". If they have to raise prices to stop hemorrhaging money, would you be willing to pay 1000 bucks a month for a max plan? Or 100$ per 1M pitput tokens (playing numberWang here, but the point stands). If I have to guess they are trying to get balance sheet in order for an IPO and they basically have 3…

$1000/mo for guaranteed functionality >= Opus 4.6 at its peak? Yes, I'd probably grumble a bit and then whip out the credit card. I'm not a heavy LLM user, and I've never come anywhere the $200/month plan limits I'm already subscribed to. But when I do use it, I want the smartest, most relentless model available, operating at the highest performance level possible. Charge what it takes to deliver that, and I'll proba…

You will, but many many many others won't do it, probably. I mean, in some parts of the world 200$ is already a big chunk of their monthly income and a price hike will definitely push them away, which is bad for the upcoming (potential) IPO.

Re: An update on recent Claude Code quality reports

#764

Post by Folcon »

giwook wrote: post #333
dbeardsl wrote: post #162

Earlier quoted context omitted.

I appreciate the reply, but I was never under the impression that gaps in conversations would increase costs nor reduce quality. Both are surprising and disappointing. I feel like that is a choice best left up to users. i.e. "Resuming this conversation with full context will consume X% of your 5-hour usage bucket, but that can be reduced by Y% by dropping old thinking logs"

Another way to think about it might be that caching is part of Anthropic's strategy to reduce costs for its users, but they are now trying to be more mindful of their costs (probably partly due to significant recent user growth as well as plans to IPO which demand fiscal prudence). Perhaps if we were willing to pay more for our subscriptions Anthropic would be able to have longer cache windows but IDK one hour seems…

The reason I've been querying the 1 hour is a user's quota resets are often longer than that, as a result I've seen situations where someone builds a large context, then hits their quota limit, waits 2+ hours, their cache is gone, their first message then eats 20%+ of their current session quota and the user doesn't want to compact as they're still trying to get the model into a good understanding of the problem, this seems to be a really painful consequence for users on anything less than a max plan which seems like an unintended consequence of Anthropic's own system design choices?

IE How their quota and caching interact with each other, it doesn't make pro and max a little different, it makes it significantly different by unintentionally penalising pro users

Re: An update on recent Claude Code quality reports

#765

Post by rekrsiv »

close04 wrote: post #667
rekrsiv wrote: post #661

Earlier quoted context omitted.

Time is finite and regression testing always gets punted to the back of the line when humans are excited. This simply reveals a staggering level of humanity.

> This simply reveals a staggering level of humanity. Wasn't AI supposed to solve all the drudgery? All those humans aided by cutting edge AI are still failing at these basic tasks? Then how good is that AI in the first place?

No, AI wasn't supposed to solve all that drudgery. The hypothesized AI singularity would, but an ordinary AI agent running an LLM is just a problem solving automaton with no will of its own, just like a fleshy brain solving computer problems is just a code monkey.

Re: An update on recent Claude Code quality reports

#766

Post by rekrsiv »

FartyMcFarter wrote: post #665
rekrsiv wrote: post #661

Earlier quoted context omitted.

Time is finite and regression testing always gets punted to the back of the line when humans are excited. This simply reveals a staggering level of humanity.

Software engineering is not a new field. Best practices on testing are mature now, and Anthropic has poached enough engineers from companies with a solid understanding of those practices. Yet, their flagship product got three really bad changes shipped into it and only resolved after more than a month. This raises another question: with all the industry-wide boasting about AI-driven productivity, why does the leading…

My theory is that most problem solvers are bad at solving problems, and most managers are bad at managing, and it doesn't matter how evolution created them: They'll make mistakes, they'll have finite time and energy, a finite context window, they'll lie and internally rewrite their own internal narratives as needed, and forget things, and drop balls, and they'll go in circles trying to find a bug they created but are too close to be able to see, and they're going to need a lot of external tooling to get through the day without forgetting anything, and constant reminders from others to get shit done. And this dynamic fundamentally creates peaks and valleys in productivity.

Wait, were we talking about humans or AI?

...

Everyone seems to be assuming either the humans or the AI has to be special. What if neither are?

Re: An update on recent Claude Code quality reports

#767

Post by xpe »

asdewqqwer wrote: post #596
xpe wrote: post #456

Earlier quoted context omitted.

To my eye, gaslighting is a serious accusation. Wikipedia's first line matches how I think of it: "Gaslighting is the manipulation of someone into questioning their perception of reality." Did I miss something? I'm only looking at primary sources to start. Not Reddit. Not The Register. Official company communications. Did Anthropic tell users i.e. "you are wrong, your experience is not worse."? If so, that would reac…

I think there are plenty of such reply on github. For example the one to AMD AI director's issue.

Please link us to it. Linking it provides an anchor for community discussion.

Re: An update on recent Claude Code quality reports

#768

Post by jghn »

troupo wrote: post #756
jghn wrote: post #752

Earlier quoted context omitted.

I’m not. Why would anyone believe marketing speak for any product? One should always assume that at best they’re fluffing their product up and more likely that they’re telling straight up lies

1. False advertisement is a thing, to the point there are laws against it 2. They were caught blatantly lying, and you're literally telling everyone it's the users' fault for not digging into the black box that is Claude Code (and more so Anthropic's servers) and figuring its behavior for themselves. A behavior that suddenly changed on a March day [1] and which previously very few people ever needed to investigate. […

I'm not saying this is a great state of affairs. But I'm saying that it's so pervasive in daily life that yes, at least part of the blame lies on users for not taking this into account. As a developer it's important to at least try to understand the tools and libraries on which one relies. Relying on magic black boxes is not a good plan on the user's part, and they need to be defensive about this. Too many developers have been more than happy to hand the keys over to the AI assistants and hope for the best.

Also it wasn't completely undocumented, rather it was hiding in not-quite-plain sight. Which itself is a bit duplicitous, but again something that's far from unique on the part of Anthropic.

Re: An update on recent Claude Code quality reports

#769

Post by xpe »

sobjornstad wrote: post #470
thomassmith65 wrote: post #446

Earlier quoted context omitted.

And still are gaslighting: We take reports about degradation very seriously. We never intentionally degrade our models [...] On March 4, we changed Claude Code's default reasoning effort from high to medium Anthropic is the best company of its kind, but that is badly worded PR.

Is adding JPEG compression to your software “intentional degradation” of the software? I wouldn't say providing a selectable option to use a faster, cheaper version of something qualifies as “degradation”. It is certainly true that they did a poor job communicating this change to users (I did not know that the default was “high” before they introduced it, I assumed they had added an effort level both above and below…

As I understand Anthropic's recent retrospective, calling the models directly via API did not change; the problem was that the harness changed and this was not communicated well to users.

Metaphorical reasoning is lossy, so talking about lossy image compression seems to be ironically fitting! ... perhaps a (hypothetical) metaphor involves Photoshop changing their default JPEG compression level without making it clear to users. PS did not change the JPEG algorithm, only a setting for it. If you look closely, you would notice it: I'll come back to this point in the last paragraph.

But a part of metaphor breaks down if you accept that Anthropic was making a net positive trade-off for customers so that they could provide a better overall service level statistically to their entire user base.

A rough metaphor for the individual versus collective trade-off might be when a retail store caps the number of toilet paper rolls customer can buy at a time. The goal is to reduce hoarding, which in a way is an analogous to Claude users having usage patterns at the high end of the statistical tail.

When it comes to PR*, transparency almost always wins? Anthropic's mistake hid the change from users, but they're going to notice when overall performance is degraded. I would hazard a guess that Claude has endured more verbal assault in the last month than in its entire history.

* both for public relations and pull requests

Re: An update on recent Claude Code quality reports

#770

Post by achierius »

xpe wrote: post #758
achierius wrote: post #668

Earlier quoted context omitted.

Do you not think people here work at big companies with big products? I do, and we have a much higher bar for shipping.

>> My overall feel is that people underestimate the complexity of the systems at Anthropic and the chaos of the growth. > Do you not think people here work at big companies with big products? I do, and we have a much higher bar for shipping. This form of comment (The "Do you not think {X}?") comes across as a swipe (discouraged by the HN guidelines). It doesn't respond to the strongest plausible interpretation of my…

That's fair. I'll adjust and say that I think there's a mix: some people certainly are bashing without understanding, but there are also a lot of engineers here whose day to day work is held to a higher standard than I think we see coming out of Anthropic, at least w.r.t. the product side of things (obviously the models are great).
Post reply on HN