Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

701–710 of 778 posts

Re: An update on recent Claude Code quality reports

#701

Post by trinsic2 »

jghn wrote: post #397
someguyiguess wrote: post #249

Earlier quoted context omitted.

Yes. It’s perfectly reasonable to expect the user to know the intricacies of the caching strategy of their llm. Totally reasonable expectation.

To some extent I'd say it is indeed reasonable. I had observed the effect for a while: if I walked away from a session I noticed that my next prompt would chew up a bunch of context. And that led me to do some digging, at which point I discovered their prompt caching. So while I'd agree with your sarcasm that expecting users to be experts of the system is a big ask, where I disagree with you is that users should be c…

Agreed. systems work the way they work. Its up to the user to determining what those limitations are. I don't like the concept of molding software based on every expectation a user has. Sometimes that expectation is unwarranted. You can see this in game development. Regardless of expressed criticism, sometimes gamers don't know what they want or what they need. A game should be developed by the design goals of the team, not cater to every whim the player base wants. We have seen were that can go.

Re: An update on recent Claude Code quality reports

#702

Post by mh- »

stickfigure wrote: post #698
tuwtuwtuwtuw wrote: post #684

Earlier quoted context omitted.

Seems like that would be trivial to test?

Most bugs are trivial to test for after you know about them.

True, but when your cache configuration has exactly 2 TTLs and modalities, I don't think it's offbase to expect them to test what happens in the cache hit/miss scenarios for each of those.

(I write this as someone who likes Claude Code, if that matters.)

Re: An update on recent Claude Code quality reports

#703

Post by pixl97 »

sfink wrote: post #674
FartyMcFarter wrote: post #665

Earlier quoted context omitted.

Software engineering is not a new field. Best practices on testing are mature now, and Anthropic has poached enough engineers from companies with a solid understanding of those practices. Yet, their flagship product got three really bad changes shipped into it and only resolved after more than a month. This raises another question: with all the industry-wide boasting about AI-driven productivity, why does the leading…

> Why does it take the company that is probably the best at agentic coding more than a month to find and solve such large regressions, even with customers complaining about them? My unfounded suspicion: because this is the tradeoff we're all facing and for the most part refusing to accept when transitioning over to LLM-driven coding. This is exactly how we're being trained to work by the strengths and limitations of…

> you might have a `--cache-dir=DIR` option that doesn't actually do anything

Working in enterprise software it's surprising how long an option that doesn't actually do anything can be missed. And that was before AI and having thousands of customers use it.

This same problem happens with documentation all the time. You end up with paragraphs or examples that simply don't reflect what the product actually does.

Re: An update on recent Claude Code quality reports

#704

Post by kovek »

vanviegen wrote: post #568
kovek wrote: post #304

Earlier quoted context omitted.

What if the cache was backed up to cold storage? Instead of having to recompute everything.

They probably already do that. But these caches can get pretty big (10s of GBs per session), so that adds up fast, even for cold storage.

10s of GBs? ( 1,000,000 context * 1,000 vector size ) ^ 2 = 1,000,000,000,000,000,000… oh wow.. I must be miscalculating

What about only storing the conversation and then recomputing the embeddings in the cache? Does that cost a lot? Doing a lot of matrix multiplication does not cost dollars of compute, especially on specialized hardware, right?

Re: An update on recent Claude Code quality reports

#705

Post by pixl97 »

prirun wrote: post #677
theshrike79 wrote: post #650

Earlier quoted context omitted.

Think of it like this: Anthropic has to keep a full virtual machine running just for you. How long should it idle there taking resources when you only pay a static monthly fee and not hourly? They have a limited number of resources and can’t keep everyone’s VM running forever.

I pay $5/mo to Vultr for a VM that runs continuously and maintains 25GB of state.

Now check out the cost difference in 25GB of computer RAM vs GPU RAM.

And yes, this is also why computer RAM has jumped the shark in costs.

The bandwidth differences in total data transferred per hour aren't even in the same 5 orders of magnitude between your server and the workloads LLMs are doing. And this is why the compute and power markets are totally screwed.

Re: An update on recent Claude Code quality reports

#707

Post by gozzoo »

TedDallas wrote: post #683
dgreensp wrote: post #658

This reveals a staggering level of incompetence, if that’s really all it is, and lack of transparency. They don’t have ANY product-level quality tests that picked this up? Many users did their own tests and published them. It’s not hard. And these users’ complaints were initially dismissed. I don’t think the high vs medium change is really on par with the others. That’s a setting you change in the UI, and depending o…

About 20 years ago I maintained a shop floor control client/server application. I asked my manager why we didn't have any independent Q/A. He said we didn't need any testers because we have 500 in the building. Wild west days then. Looks like we are back.

It is worse than that. People have been complaining for weeks and Anthropic’s message was basically “you are holding it wrong”. On top of that this misconfiguration somehow makes CC consume much more tokens. How believable is all that?

Re: An update on recent Claude Code quality reports

#708

Post by pixl97 »

margalabargala wrote: post #300
computably wrote: post #178

Earlier quoted context omitted.

> I was never under the impression that gaps in conversations would increase costs nor reduce quality. Both are surprising and disappointing. You didn't do your due diligence on an expensive API. A naïve implementation of an LLM chat is going to have O(N^2) costs from prompting with the entire context every time. Caching is needed to bring that down to O(N), but the cache itself takes resources, so evictions have to…

Okay, sure. There's a dollar/intelligence tradeoff. Let me decide to make it, don't silently make Claude dumber because I forgot about a terminal tab for an hour. Just because a project isn't urgent doesn't mean it's not important. If I thought it didn't need intelligence I would use Sonnet or Haiku.

"Gets mad because their is no option"

"Gets mad because when their is options the defaults suck"

"Gets mad because the options start massively increasing costs to areospace pricing"

Re: An update on recent Claude Code quality reports

#709

Post by pixl97 »

charcircuit wrote: post #534
mpyne wrote: post #355

Earlier quoted context omitted.

The trace goes back fine, that's not the issue. The issue is that if they send the full trace back, it will have to be processed from the start if the cache expired, and doing that will cause a huge one-time hit against your token limit if the session has grown large. So what Boris talked about is stripping things out of the trace that goes back to regenerate the session if the cache expires. Doing this would help av…

>and doing that will cause a huge one-time hit against your token limit if the session has grown large. Anthropic already profited from generating those tokens. They can afford subsidize reloading context.

No they can't, that's what you don't seem to get.

Reloading those tokens takes around the same effort as processing them in the first place.

It's ok to be ignorant of how the infrastructure for LLMs work, just don't be proud of it.

Re: An update on recent Claude Code quality reports

#710

Post by Majromax »

kovek wrote: post #704
vanviegen wrote: post #568

Earlier quoted context omitted.

They probably already do that. But these caches can get pretty big (10s of GBs per session), so that adds up fast, even for cold storage.

10s of GBs? ( 1,000,000 context * 1,000 vector size ) ^ 2 = 1,000,000,000,000,000,000… oh wow.. I must be miscalculating What about only storing the conversation and then recomputing the embeddings in the cache? Does that cost a lot? Doing a lot of matrix multiplication does not cost dollars of compute, especially on specialized hardware, right?

Context length 1e6, vector length 1e3, and 1e2 model layers for 100e9 context size. Costs will go up even more with a richer latent space and more model layers, and the western frontier outfits are reasonably likely to be maximizing both.
Post reply on HN