Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

751–760 of 778 posts

Re: An update on recent Claude Code quality reports

#751

Post by bmitc »

bcherny wrote: post #124
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

Appreciate the responses here. However, I feel like these responses are just to show us how much you know about the product and aren't actually helpful.

Instead, why don't you and Anthropic be more open about changes to these tools rather than waiting for users to complain, then investigating things after the fact that you should have investigated in the first place, and then posting on social media about all the cool tech details?

My company is tens of thousands strong. The amount of churn in Claude Code is a major issue and causing real awareness of the lack of stability and lack of customer support Anthropic provides.

And Claude Code is actually becoming a prototypical example of the dangers of vibe coded products and the burdens they place.

Re: An update on recent Claude Code quality reports

#752

Post by jghn »

taormina wrote: post #745
jghn wrote: post #735

Earlier quoted context omitted.

My point is the opposite. I don't think my observation was smart, and I'm surprised to so many people here, a venue with a lot of people who use this stuff far more than I do, think it wasn't an easy to grok thing.

You’re still intentionally missing the point. Everyone knows they are lying. It doesn’t excuse the lies!

I’m not. Why would anyone believe marketing speak for any product? One should always assume that at best they’re fluffing their product up and more likely that they’re telling straight up lies

Re: An update on recent Claude Code quality reports

#753

Post by bavell »

andrewingram wrote: post #649
bavell wrote: post #638

Earlier quoted context omitted.

> As a user, I _expect_ the cost of resuming X hours/days later to be no different to resuming seconds or minutes later. As an informed user who understands his tools, I of course expect large uncached conversations to massively eat into my token budget, since that's how all of the big LLM providers work. I also understand these providers are businesses trying to make money and they aren't going to hold every convers…

I'd hazard a guess that there's a large gulf between proportion of users who know as much as you, and the total number using these tools. The fact that a message can perform wildly differently (in either cost, or behaviour if using one of the mitigations) based on whether I send it at t vs t+1 seems like a major UX issue, especially given t is very likely not exposed in the UI.

I definitely agree that it should be shown and obvious in the UI. They do show a warning now when resuming old sessions but still could be better.

Re: An update on recent Claude Code quality reports

#754

Post by generalpf »

sfink wrote: post #674
FartyMcFarter wrote: post #665

Earlier quoted context omitted.

Software engineering is not a new field. Best practices on testing are mature now, and Anthropic has poached enough engineers from companies with a solid understanding of those practices. Yet, their flagship product got three really bad changes shipped into it and only resolved after more than a month. This raises another question: with all the industry-wide boasting about AI-driven productivity, why does the leading…

> Why does it take the company that is probably the best at agentic coding more than a month to find and solve such large regressions, even with customers complaining about them? My unfounded suspicion: because this is the tradeoff we're all facing and for the most part refusing to accept when transitioning over to LLM-driven coding. This is exactly how we're being trained to work by the strengths and limitations of…

>There'll probably be a startup tomorrow that digests user input and boils out the noise to provide a robust enough signal to guide some monitoring agent, and it'll help some cases, and train us to be even worse at others.

This sounds like Enterpret.

Re: An update on recent Claude Code quality reports

#755

Post by 6keZbCECT2uB »

bcherny wrote: post #124
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

I can see how this makes sense as a default behavior for cost conscious users. I would prefer to have the option for my company to pay more to rehydrate the cache than to have there be a model performance difference when having idled for an hour.

"We tried a few different approaches to improve this UX:

1. Educating users on X/social

2. Adding an in-product tip to recommend running /clear when re-visiting old conversations (we shipped a few iterations of this)

3. Eliding parts of the context after idle: old tool results, old messages, thinking. Of these, thinking performed the best, and when we shipped it, that's when we unintentionally introduced the bug in the blog post."

I see how these interventions help users reduce their token burn rate, but they don't address the need for an enterprise user to maintain quality.

A common workflow for me is kick off a prompt, commute home, eat dinner, follow up on prompt. Frequently 80K tokens or less in the context, frequently > 3 hours. Or when running multiple sessions it's easy to let a session idle for a few hours while I focus on one. Or many meetings might mean idle time for an hour.

Also, for enterprise users, I don't think education on X is a great place. There are people upskilling on this that never intentionally go on X.

First thing that comes to mind would be a weekly tip feed of footguns and underutilized functionality published to an anthropic website. "The Old New Thing" "Guru of the Week" "Abseil tips of the week" all have that format.

Re: An update on recent Claude Code quality reports

#756

Post by troupo »

jghn wrote: post #752
taormina wrote: post #745

Earlier quoted context omitted.

You’re still intentionally missing the point. Everyone knows they are lying. It doesn’t excuse the lies!

I’m not. Why would anyone believe marketing speak for any product? One should always assume that at best they’re fluffing their product up and more likely that they’re telling straight up lies

1. False advertisement is a thing, to the point there are laws against it

2. They were caught blatantly lying, and you're literally telling everyone it's the users' fault for not digging into the black box that is Claude Code (and more so Anthropic's servers) and figuring its behavior for themselves. A behavior that suddenly changed on a March day [1] and which previously very few people ever needed to investigate.

[1] https://x.com/levelsio/status/2029307862493618290

Re: An update on recent Claude Code quality reports

#757

Post by coldtea »

kang wrote: post #360
coldtea wrote: post #321

Earlier quoted context omitted.

It seems you haven't done the due diligence on what the parent meant :) It's not about "constructing a prompt" in the sense of building the prompt string. That of course wouldn't be costly. It is about reusing llm inference state already in GPU memory (for the older part of the prompt that remains the same) instead of rerunning the prompt and rebuilding those attention tensors from scratch.

You not only skipped the diligence but confused everyone repeating what I said :( that is what caching is doing. the llm inference state is being reused. (attention vectors is internal artefact in this level of abstraction, effectively at this level of abstraction its a the prompt). The part of the prompt that has already been inferred no longer needs to be a part of the input, to be replaced by the inference subset.…

>It seems you haven't done the due diligence on what part of the API is expensive - constructing a prompt shouldn't be same charge/cost as llm pass.

I think you missed what the parent meant then, and the confusing way you replied seemed to imply that they're not doing inference caching (the opposite of what you wanted to mean).

The parent didn't said that caching is needed to merely avoid reconstructing the prompt as string. He just takes that for granted that it means inference caching, to avoid starting the session totally new. That's how I read "from prompting with the entire context every time" (not the mere string).

So when you answered as if they're wrong, and wrote "constructing a prompt shouldn't be same charge/cost as llm pass", you seemed to imply "constructing a prompt shouldn't be same charge/cost as llm pass [but due to bad implementation or overcharging it is]".

Re: An update on recent Claude Code quality reports

#758

Post by xpe »

achierius wrote: post #668
xpe wrote: post #443

Earlier quoted context omitted.

> All of this points to their priorities not being aligned with their users’. Framing this as "aligned" or "not aligned" ignores the interesting reality in the middle. It is banal to say an organization isn't perfectly aligned with its customers. I'm not disagreeing with the commenter's frustration. But I think it can help to try something out: take say the top three companies whose product you interact with on a reg…

Do you not think people here work at big companies with big products? I do, and we have a much higher bar for shipping.

>> My overall feel is that people underestimate the complexity of the systems at Anthropic and the chaos of the growth.

> Do you not think people here work at big companies with big products? I do, and we have a much higher bar for shipping.

This form of comment (The "Do you not think {X}?") comes across as a swipe (discouraged by the HN guidelines). It doesn't respond to the strongest plausible interpretation of my comment (also in the guidelines).

Re: An update on recent Claude Code quality reports

#759

Post by willis936 »

xpe wrote: post #740
willis936 wrote: post #734

Earlier quoted context omitted.

I didn't assume bad faith, I simply reworded your conclusions with less soft language so that others would understand your position more clearly. You are saying what they are doing is hard. That's fine. Their stated goals are to be the responsible stewards of the technology and we agree they are failing at that goal. You would attribute that to incompetence and not malice.

I personally try to follow Rapoport's Rules, and I since think they are consistent with the HN Guidelines, I like to mention them: [1]. I've thought on it, and I will try to start off with something we both agree on... We both agree that Anthropic made some mistakes, but this is probably a pretty uninteresting and shallow agreement. I find it unlikely that we would enumerate or characterize the mistakes similarly. I…

Brevity is a virtue. Focus on the cooperative principle.

Re: An update on recent Claude Code quality reports

#760

Post by sfink »

alfons_foobar wrote: post #750
sfink wrote: post #674

Earlier quoted context omitted.

> Why does it take the company that is probably the best at agentic coding more than a month to find and solve such large regressions, even with customers complaining about them? My unfounded suspicion: because this is the tradeoff we're all facing and for the most part refusing to accept when transitioning over to LLM-driven coding. This is exactly how we're being trained to work by the strengths and limitations of…

I am not sure this approach can take you very far. In my experience, CC makes it very very easy to _add_ things, resulting in much more code / features. CC can obviously read/understand a codebase much faster than we do, but this also has a limit (how much context we can feed into it) - I think your approch is in essence a bet that future models' ability to read/understand code (size of context) improves as fast or f…

Ouch. I guess this came across as "my approach". I haven't done enough agentic coding to feel like I know enough to have a worthwhile, but at the moment I'm squarely in your camp. I don't believe it's going to work to let an agent loose expanding a teetering codebase with little to no concern for maintainability. We're going to have to painfully relearn the lessons of pre-AI coding, whatever that means with AI in the mix.
Post reply on HN