Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

501–510 of 778 posts

Re: An update on recent Claude Code quality reports

#501

Post by abustamam »

Octoth0rpe wrote: post #438
abustamam wrote: post #412

Earlier quoted context omitted.

> users should be curious and actively attempting to understand how it works Have you ever talked with users? > this is an endless job Indeed. If we spend all our time learning what changed with all our tooling when it changes without proper documentation then we spend all our working lives keeping up instead of doing our actual jobs.

There are general users of the average SaaS, and there are claude code users. There's no doubt in my mind that our expectations should be somewhat higher for CC users re: memory. I'm personally not completely convinced that cache eviction should be part of their thought process while using CC, but it's not _that_ much of a stretch.

Personally I've never thought about cache eviction as it pertains to CC. It's just not something that I ever needed to think about. Maybe I'm just not a power user but I just use the product the way I want to and it just works.

Re: An update on recent Claude Code quality reports

#502

Post by oofbey »

xpe wrote: post #456
thomassmith65 wrote: post #446

Earlier quoted context omitted.

And still are gaslighting: We take reports about degradation very seriously. We never intentionally degrade our models [...] On March 4, we changed Claude Code's default reasoning effort from high to medium Anthropic is the best company of its kind, but that is badly worded PR.

To my eye, gaslighting is a serious accusation. Wikipedia's first line matches how I think of it: "Gaslighting is the manipulation of someone into questioning their perception of reality." Did I miss something? I'm only looking at primary sources to start. Not Reddit. Not The Register. Official company communications. Did Anthropic tell users i.e. "you are wrong, your experience is not worse."? If so, that would reac…

They didn’t say “your experience is not worse” but they did frequently say “just turn reasoning effort back up and it will be fine”. And that pretty explicitly invalidates all the (correct) feedback which said it’s not just reasoning effort.

They knew they had deliberately made their system worse, despite their lame promise published today that they would never do such a thing. And so they incorrectly assumed that their ham fisted policy blunder was the only problem.

Still plenty I prefer about Claude over GPT but this really stings.

Re: An update on recent Claude Code quality reports

#503

Post by nozzlegear »

slashdave wrote: post #398
cmrdporcupine wrote: post #372

Earlier quoted context omitted.

They're losing customers because of quality concerns. Pausing development and focusing 100% on quality is how you fix that. That said, that may not have been obvious at all in the Jan/Feb time frame when they got a wave of customers due to ethical concerns.

No. Pausing development does not make compute (you know, physical machines?) appear out of thin air.

On the other hand, sacrificing your paying customers at the altar of compute and tokens does not make money appear out of thin air.

Re: An update on recent Claude Code quality reports

#504

Post by bontaq »

jannyfer wrote: post #390
bontaq wrote: post #352

Earlier quoted context omitted.

How's that O(N^2)? How's it O(N) with caching? Does a 3 turn conversation cost 3 times as much with no caching, or 9 times as much?

I’m not sure that it’s O(N) with caching but this illustrates the N^2 part: https://blog.exe.dev/expensively-quadratic

If there was an exponential cost, I would expect to see some sort of pricing based on that. I would also expect to see it taking exponentially longer to process a prompt. I don't believe LLMs work like that. The "scary quadratic" referenced in what you linked seems to be pointing out that cache reads increase as your conversation continues?

If I'm running a database keeping track of a conversation, and each time it writes the entire history of the conversation instead of appending a message, are we calling that O(N^2) now?

Re: An update on recent Claude Code quality reports

#505

Post by cortesoft »

matheusmoreira wrote: post #498
vyr wrote: post #408

Earlier quoted context omitted.

a countdown clock telling you that you should talk to the model again before your streak expires? that's the kind of UX i'd expect from an F2P mobile game or an abandoned shopping cart nag notification

Why not an automated ping message that's cheap for the model to respond to?

Because the cache is held on anthropics side, and they aren't going to hold your context in cache indefinitely.

Re: An update on recent Claude Code quality reports

#506

Post by cortesoft »

raron wrote: post #261
computably wrote: post #178

Earlier quoted context omitted.

> I was never under the impression that gaps in conversations would increase costs nor reduce quality. Both are surprising and disappointing. You didn't do your due diligence on an expensive API. A naïve implementation of an LLM chat is going to have O(N^2) costs from prompting with the entire context every time. Caching is needed to bring that down to O(N), but the cache itself takes resources, so evictions have to…

How big this cached data is? Wouldn't it be possible to download it after idling a few minutes "to suspend the session", and upload and restore it when the user starts their next interaction?

What they mean when they say 'cached' is that it is loaded into the GPU memory on anthropic servers.

You already have the data on your own machine, and that 'upload and restore' process is exactly what is happening when you restart an idle session. The issue is that it takes time, and it counts as token usage because you have to send the data for the GPU to load, and that data is the 'tokens'.

Re: An update on recent Claude Code quality reports

#507

Post by leobuskin »

This usage reset you did on April 23 will not mitigate the struggle we’ve experienced. I didn’t even notice it yesterday. I checked this morning and it came down from 25% weekly to 7%. What is this? I didn’t have problems for two months like many others (maybe my CC habits helped), but two weeks were very painful. Make a proper apology, guys. This “reset” for many users could hit the first days of the week, tell me you thought about that.

Re: An update on recent Claude Code quality reports

#508

Post by MrOrelliOReilly »

joshribakoff wrote: post #388
MrOrelliOReilly wrote: post #317

IMO this is the consequence of a relentless focus on feature development over core product refinement. I often have the impression that Anthropic would benefit from a few senior product people. Someone needs to lend them a copy of “Escaping the Build Trap.” Just because we _can_ rapidly add features now doesn’t mean we should. PS I’m not referencing a well-known book to suggest the solution is trite product group thi…

They had like 100 devs making 600k at one point. The issue is certainly not lack of talent. More like, they insist on forcing the vibe coding narrative. Some candidates are refusing interview requests accordingly.

Ugh wrote “latter” and meant “former.” I didn’t mean lack of eng talent, but product

Re: An update on recent Claude Code quality reports

#510

Post by ElFitz »

szmarczak wrote: post #421
ElFitz wrote: post #307

Earlier quoted context omitted.

By imposing the use of their harness, they control the system prompt: > On April 16, we added a system prompt instruction to reduce verbosity. In combination with other prompt changes, it hurt coding quality, and was reverted on April 20. This impacted Sonnet 4.6, Opus 4.6, and Opus 4.7 They can pick the default reasoning effort: > On March 4, we changed Claude Code's default reasoning effort from high to medium to r…

> they control the system prompt They control the default system prompt. You can change it if you want to. > They can pick the default reasoning effort Don't see how it's an obstacle in allowing third party wrappers. > They can decide what to keep and what to throw out That's actually a good point. However I still don't think it's an obstacle. If third party wrappers were bad, people simply wouldn't be using them.

Evidently, all these things you just dismissed matter, else all the changes I quoted from the original post wouldn’t have affected anyone, or half as many people, or half as much. Anthropic wouldn’t have had any complaints to investigate, the article promoting this entire thread wouldn’t exist, and we wouldn’t be having this very conversation.

Defaults matter. A large share of people never change them (status quo bias, psychological inertia). Having control over them (and usage quotas) means Anthropic can control and fine-tune what this fixed subscription costs them.

And evidently (re, the original article), they tried to do so.

Post reply on HN