Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

421–430 of 778 posts

Re: An update on recent Claude Code quality reports

#421

Post by szmarczak »

ElFitz wrote: post #307
szmarczak wrote: post #204

Earlier quoted context omitted.

Nothing you wrote makes sense. The limits are so Anthropic isn't on a loss. If they can customize Claude using Code, I see no reason why they couldn't do so with other wrappers. Other wrappers can also make use of cache. If you worry about "degraded" experience, then let people choose. People won't be using other wrappers if they turn out to be bad. People ain't stupid.

By imposing the use of their harness, they control the system prompt: > On April 16, we added a system prompt instruction to reduce verbosity. In combination with other prompt changes, it hurt coding quality, and was reverted on April 20. This impacted Sonnet 4.6, Opus 4.6, and Opus 4.7 They can pick the default reasoning effort: > On March 4, we changed Claude Code's default reasoning effort from high to medium to r…

> they control the system prompt

They control the default system prompt. You can change it if you want to.

> They can pick the default reasoning effort

Don't see how it's an obstacle in allowing third party wrappers.

> They can decide what to keep and what to throw out

That's actually a good point. However I still don't think it's an obstacle. If third party wrappers were bad, people simply wouldn't be using them.

Re: An update on recent Claude Code quality reports

#422

Post by FuckButtons »

slashdave wrote: post #362
Joeri wrote: post #180

Earlier quoted context omitted.

This sounds like one of those problems where the solution is not a UX tweak but an architecture change. Perhaps prompt cache should be made long term resumable by storing it to disk before discarding from memory?

Disk where? LLM requests are routed dynamically. You might not even land in the same data center.

But if you have a tiered cache, then waiting several seconds / minutes is still preferable to getting a cache miss. I suspect the larger problem is the amount of tinkering they are doing with the model makes that not viable.

Re: An update on recent Claude Code quality reports

#423

Post by FuckButtons »

8note wrote: post #264
bcherny wrote: post #124

Earlier quoted context omitted.

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

reasonably, if i'm in an interactive session, its going to have breaks for an hour or more. whats driving the hour cache? shouldnt people be able to have lunch, then come back and continue? are you expecting claude code users to not attend meetings? I think product-wise you might need a better story on who uses claude-code, when and why. Same thing with session logs actually - i know folks who are definitely going to…

As with everything Anthropic recently this is a supply constraint issue. They have not planned for scale adequately.

Re: An update on recent Claude Code quality reports

#425

Post by epsteingpt »

cmenge wrote: post #401

Bit surprised about the amount of flak they're getting here. I found the article seemed clear, honest and definitely plausible. The deterioration was real and annoying, and shines a light on the problematic lack of transparency of what exactly is going on behind the scenes and the somewhat arbitrary token-cost based billing - too many factors at play, if you wanted to trace that as a user you can just do the work you…

They gaslit people for months saying it wasn't an issue publicly.

That's the reason for the flak

Re: An update on recent Claude Code quality reports

#426

Post by chris1993 »

bcherny wrote: post #124
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

So this explains why resuming a session after a 5-hour timeout basically eats most of the next session. How then to avoid this?

Re: An update on recent Claude Code quality reports

#427

Post by epsteingpt »

podnami wrote: post #77

They lost me at Opus 4.7 Anecdotally OpenAI is trying to get into our enterprise tooth and nail, and have offered unlimited tokens until summer. Gave GPT5.4 a try because of this and honestly I don’t know if we are getting some extra treatment, but running it at extra high effort the last 30 days I’ve barely see it make any mistakes. At some points even the reasoning traces brought a smile to my face as it preemptive…

Truth

Re: An update on recent Claude Code quality reports

#428

Post by cadamsdotcom »

sharts wrote: post #382
giwook wrote: post #333

Earlier quoted context omitted.

Another way to think about it might be that caching is part of Anthropic's strategy to reduce costs for its users, but they are now trying to be more mindful of their costs (probably partly due to significant recent user growth as well as plans to IPO which demand fiscal prudence). Perhaps if we were willing to pay more for our subscriptions Anthropic would be able to have longer cache windows but IDK one hour seems…

That doesn’t make sense to pay more for cache warming. Your session for the most part is already persisted. Why would it be reasonable to pay again to continue where you left off at any time in the future?

Sure, it wouldn’t make sense if they only had one customer to serve :)

Re: An update on recent Claude Code quality reports

#429

Post by jimkleiber »

JumpCrisscross wrote: post #179
dbeardsl wrote: post #162

Earlier quoted context omitted.

I appreciate the reply, but I was never under the impression that gaps in conversations would increase costs nor reduce quality. Both are surprising and disappointing. I feel like that is a choice best left up to users. i.e. "Resuming this conversation with full context will consume X% of your 5-hour usage bucket, but that can be reduced by Y% by dropping old thinking logs"

> I was never under the impression that gaps in conversations would increase costs The UI could indicate this by showing a timer before context is dumped.

I tried to hack the statusline to show this but when i tried, i don't think the api gave that info. I'd love if they let us have more variables to access in the statusline.

Re: An update on recent Claude Code quality reports

#430

Post by websap »

mpyne wrote: post #344
solarkraft wrote: post #246

Earlier quoted context omitted.

I somewhat disagree that this is due diligence. Claude Code abstracts the API, so it should abstract this behavior as well, or educate the user about it.

> Claude Code abstracts the API, so it should abstract this behavior as well, or educate the user about it. Does mmap(2) educate the developer on how disk I/O works? At some point you have to know something about the technology you're using, or accept that you're a consumer of the ever-shifting general best practice, shifting with it as the best practice shifts.

Does using print() in Python means I need to understand the Kernel? This is an absurd thought.
Post reply on HN