Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

Post reply on HN
Print view View original

311–320 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#311

Post by varunsrinivas »

trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Thanks for the clarification. When you say “harness issue,” does that mean the problem was in the Claude Code wrapper / execution environment rather than the underlying model itself?

Curious whether this affected things like prompt execution order, retries, or tool calls, or if it was mostly around how requests were being routed. Understanding the boundary would help when debugging similar setups.

Re: Claude Code daily benchmarks for degradation tracking

#312

Post by threethirtytwo »

Does this even make sense? Clearly anthropic won't release a model unless it passed a benchmark of some sort that proves it's better than the previous model... or else why would they even release it?

It's obvious if this thing shows degradation, than there is another thing that is showing improvement.

Re: Claude Code daily benchmarks for degradation tracking

#315

Post by TOMDM »

mathrawka wrote: post #294
samlinnfer wrote: post #269

Earlier quoted context omitted.

Is there compensation for the tokens because Claude wasted all of them?

You are funny. Anthropic refuses to issue refunds, even when they break things. I had an API token set via an env var on my shell, and claude code changed to read that env var. I had a $10 limit set on it, so found out it was using the API, instead of my subscription, when it stopped working. I filed a ticket and they refused to refund me, even though it was a breaking change with claude code.

Anthropic just reduced the price of the team plan and refunded us on the prior invoice.

YMMV

Re: Claude Code daily benchmarks for degradation tracking

#317

Post by Ekaros »

trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Why wasn't this change review by infallible AI? How come an AI company that now must be using more advanced AI than anyone else would allow this happen?

Re: Claude Code daily benchmarks for degradation tracking

#318

Post by nikanj »

someguyiguess wrote: post #301
jonaustin wrote: post #270

Earlier quoted context omitted.

How about how Claude 2.1.x is "literally unusable" because it frequently completely hangs (requires kill -9) and uses 100% cpu? https://github.com/anthropics/claude-code/issues/18532

What OS? Does this happen randomly, after long sessions, after context compression? Do you have any plugins / mcp servers running? I used to have this same issue almost every session that lasted longer than 30 minutes. It seemed to be related to Claude having issues with large context windows. It stopped happening maybe a month ago but then I had it happen again last week. I realized it was due to a third-party mcp s…

Windows with no plugins and my Claude is exactly like this

Re: Claude Code daily benchmarks for degradation tracking

#319

Post by debugnik »

swyx wrote: post #194
Dolores12 wrote: post #39

Earlier quoted context omitted.

so basically they know requests using your API key should be treated with care?

they could but you can also have some trust in anthropic to have some integrity there, these are earnest people. "trust but verify" ofc . https://latent.space/p/artificialanalysis do api keys but also mystery shopper checks

That's why we're setting up adversarial benchmarks to test if they are doing the thing they promised not to do, because we totally trust them.
Post reply on HN