Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on recent Claude Code quality reports

anthropic.com

Post reply on HN
Print view View original

141–150 of 778 posts

Re: An update on recent Claude Code quality reports

#141

Post by arjie »

dataviz1000 wrote: post #38

This is the problem with co-opting the word "harness". What agents need is a test harness but that doesn't mean much in the AI world. Agents are not deterministic; they are probabilistic. If the same agent is run it will accomplish the task a consistent percentage of the time. I wish I was better at math or English so I could explain this. I think they call it EVAL but developers don't discuss that too much. All they…

The word is not co-opted. A harness is just supportive scaffolding to run something. A test harness is scaffolding to run tests against software, a fuzz harness is scaffolding to run a fuzzer against the software, and so on. I've seen it being used in this manner many times over the past 15 years. It's the device that wraps your software so you can run it repeatedly with modifications of parameters, source code, or test condition.

Re: An update on recent Claude Code quality reports

#142

Post by qsort »

fidrelity wrote: post #128
bcherny wrote: post #124

Earlier quoted context omitted.

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

Just wanted to say I appreciate your responses here. Engaging so directly with a highly critical audience is a minefield that you're navigating well. Thank you.

I agree with this.

I'm writing this message even though I don't have much to add because it's often the case on HN that criticism is vocal and appreciation is silent and I'd like to balance out the sentiment.

Anthropic has fumbled on many fronts lately but engaging honestly like this is the right thing to do. I trust you'll get back on track.

Re: An update on recent Claude Code quality reports

#143

Post by gverrilla »

bcherny wrote: post #124
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

I drop sessions very frequently to resume later - that's my main workflow with how slow Claude is. Is there anything I can do to not encounter this cache problem?

Re: An update on recent Claude Code quality reports

#146

Post by troupo »

fidrelity wrote: post #128
bcherny wrote: post #124

Earlier quoted context omitted.

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

Just wanted to say I appreciate your responses here. Engaging so directly with a highly critical audience is a minefield that you're navigating well. Thank you.

> Engaging so directly with a highly critical audience is a minefield that you're navigating well.

They spent two months literally gaslighting this "critical audience" that this could not be happening and literally blaming users for using their vibe-coded slop exactly as advertised.

All the while all the official channels refused to acknowledge any problems.

Now the dissatisfaction and subscription cancellations have reached a point where they finally had to do something.

Re: An update on recent Claude Code quality reports

#147

Post by walthamstow »

ctoth wrote: post #135

> As of April 23, we’re resetting usage limits for all subscribers. Wait, didn't they just reset everybody's usage last Thursday, thereby syncing everybody's windows up? (Mine should have reset at 13:00 MDT) ? So this is just the normal weekly reset? Except now my reset says it will come Saturday? This is super-confusing!

The weekly reset point is different per account. I think something to do with first sign-up date. Mine is on a Tuesday.

Re: An update on recent Claude Code quality reports

#148

Post by isaacdl »

bcherny wrote: post #124
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

Thanks for giving more information. Just as a comment on (1), a lot of people don't use X/social. That's never going to be a sustainable path to "improve this UX" since it's...not part of the UX of the product.

It's a little concerning that it's number 1 in your list.

Re: An update on recent Claude Code quality reports

#149

Post by scuderiaseb »

davidfstr wrote: post #100

Good on Anthropic for giving an update & token refund, given the recent rumors of an inexplicable drop in quality. I applaud the transparency.

Opus 4.7 was released a week ago, at that point all limits were reset, so this was very beneficial to them because basically everyones weekly limit Was anyway about to be reset.

Re: An update on recent Claude Code quality reports

#150

Post by ceuk »

bcherny wrote: post #124
6keZbCECT2uB wrote: post #61

"On March 26, we shipped a change to clear Claude's older thinking from sessions that had been idle for over an hour, to reduce latency when users resumed those sessions. A bug caused this to keep happening every turn for the rest of the session instead of just once, which made Claude seem forgetful and repetitive. We fixed it on April 10. This affected Sonnet 4.6 and Opus 4.6" This makes no sense to me. I often leav…

Hey, Boris from the Claude Code team here. Normally, when you have a conversation with Claude Code, if your convo has N messages, then (N-1) messages hit prompt cache -- everything but the latest message. The challenge is: when you let a session idle for >1 hour, when you come back to it and send a prompt, it will be a full cache miss, all N messages. We noticed that this corner case led to outsized token costs for u…

Is having massive sessions which sit idle for hours (or days) at a time considered unusual? That's a really, really common scenario for me.

Two questions if you see this:

1) if this isn't best practice, what is the best way to preserve highly specific contexts?

2) does this issue just affect idle sessions or would the cache miss also apply to /resume ?

Post reply on HN