Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude Code daily benchmarks for degradation tracking

marginlab.ai

Post reply on HN
Print view View original

321–330 of 372 posts

Re: Claude Code daily benchmarks for degradation tracking

#321

Post by xnorswap »

isaacdl wrote: post #206
trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Anywhere we can read more about what a "harness issue" means? What was the impact of it?

One thing that could be a strong degradation especially for benchmarks is they switched the default "Exit Plan" mode from:

    "Proceed"
to

   "Clear Context and Proceed"
It's rare you'd want to do that unless you're actually near the context window after planning.

I pressed it accidentally once, and it managed to forget one of the clarifying questions it asked me because it hadn't properly written that to the plan file.

If you're running in yolo mode ( --dangerously-skip-permissions ) then it wouldn't surprise me to see many tasks suddenly do a lot worse.

Even in the best case, you've just used a ton of tokens searching your codebase, and it then has to repeat all that to implement because it's been cleared.

I'd like to see the option of:

    "Compact and proceed"
because that would be useful, but just proceed should still be the default imo.

Re: Claude Code daily benchmarks for degradation tracking

#322

Post by asimovDev »

svdr wrote: post #50
silverlight wrote: post #19

There was a moment about a week ago where Claude went down for about an hour. And right after it came back up it was clear a lot of people had given up and were not using it. It was probably 3x faster than usual. I got more done in the next hour with it than I do in half a day usually. It was definitely a bit of a glimpse into a potential future of “what if these things weren’t resource constrained and could just fly…

I would also regret it if they become that fast; right now I can really take a moment to enjoy the hard work the model is doing for me.

https://xkcd.com/303/

the evolution of this xkcd

Re: Claude Code daily benchmarks for degradation tracking

#323

Post by PeterStuer »

selcuka wrote: post #283
easygenes wrote: post #264

Earlier quoted context omitted.

He doesn't: https://x.com/trq212/status/2014051501786931427

For those who don't want to visit X: Most people's mental model of Claude Code is that "it's just a TUI" but it should really be closer to "a small game engine". For each frame our pipeline constructs a scene graph with React then -> layouts elements -> rasterizes them to a 2d screen -> diffs that against the previous screen -> finally uses the diff to generate ANSI sequences to draw We have a ~16ms frame budget so w…

This is just the sort of bloated overcomplication I often see in first iteration AI generated solutions before I start pushing back to reduce the complexity.

Usually, after 4-5 iterations, you can get something that has shed 80-90% of the needless overcomplexification.

My personal guess is this is inherent in the way LLMs integrate knowledge during training. You always have a tradeoff in contextualization vs generalization.

So the initial response is often a plugged together hack from 5 different approaches, your pushbacks provide focus and constraints towards more inter-aligned solution approaches.

Re: Claude Code daily benchmarks for degradation tracking

#324

Post by mvandermeulen »

samlinnfer wrote: post #269
trq_ wrote: post #191

Hi everyone, Thariq from the Claude Code team here. Thanks for reporting this. We fixed a Claude Code harness issue that was introduced on 1/26. This was rolled back on 1/28 as soon as we found it. Run `claude update` to make sure you're on the latest version.

Is there compensation for the tokens because Claude wasted all of them?

You’re lucky they have even admitted a problem instead of remaining silent and quietly fixing it. Do not expect ethical behaviour from this company.

Re: Claude Code daily benchmarks for degradation tracking

#325

Post by gmueckl »

jmalicki wrote: post #236
gmueckl wrote: post #217

Earlier quoted context omitted.

No, this isn't right. There are totally legitimate use cases for PRNGs as sources of random number sequences following a certain probability distribution where freezing the seed and getting reproducibility is actually required.

And for a complicated concurrent system you can also replay the exact timings and orderings as well!

That's completely different from PRNGs. I don't understand why you think those things belong together.

Re: Claude Code daily benchmarks for degradation tracking

#326

Post by ttoinou »

someguyiguess wrote: post #302
selcuka wrote: post #283

Earlier quoted context omitted.

For those who don't want to visit X: Most people's mental model of Claude Code is that "it's just a TUI" but it should really be closer to "a small game engine". For each frame our pipeline constructs a scene graph with React then -> layouts elements -> rasterizes them to a 2d screen -> diffs that against the previous screen -> finally uses the diff to generate ANSI sequences to draw We have a ~16ms frame budget so w…

Interesting. On first glance that seems over engineered. I wonder what the reason is for doing it that way?

Vibecoded ?

Re: Claude Code daily benchmarks for degradation tracking

#327

Post by Kelteseth »

someguyiguess wrote: post #302
selcuka wrote: post #283

Earlier quoted context omitted.

For those who don't want to visit X: Most people's mental model of Claude Code is that "it's just a TUI" but it should really be closer to "a small game engine". For each frame our pipeline constructs a scene graph with React then -> layouts elements -> rasterizes them to a 2d screen -> diffs that against the previous screen -> finally uses the diff to generate ANSI sequences to draw We have a ~16ms frame budget so w…

Interesting. On first glance that seems over engineered. I wonder what the reason is for doing it that way?

Claude made it /s

Re: Claude Code daily benchmarks for degradation tracking

#328

Post by rubslopes »

xnorswap wrote: post #321
isaacdl wrote: post #206

Earlier quoted context omitted.

Anywhere we can read more about what a "harness issue" means? What was the impact of it?

One thing that could be a strong degradation especially for benchmarks is they switched the default "Exit Plan" mode from: "Proceed" to "Clear Context and Proceed" It's rare you'd want to do that unless you're actually near the context window after planning. I pressed it accidentally once, and it managed to forget one of the clarifying questions it asked me because it hadn't properly written that to the plan file. If…

Not disagreeing with you, but FYI you can roll back to the conversation before the 'clear context and proceed' with 'claude --resume'.

Re: Claude Code daily benchmarks for degradation tracking

#329

Post by mike_hearn »

someguyiguess wrote: post #302
selcuka wrote: post #283

Earlier quoted context omitted.

For those who don't want to visit X: Most people's mental model of Claude Code is that "it's just a TUI" but it should really be closer to "a small game engine". For each frame our pipeline constructs a scene graph with React then -> layouts elements -> rasterizes them to a 2d screen -> diffs that against the previous screen -> finally uses the diff to generate ANSI sequences to draw We have a ~16ms frame budget so w…

Interesting. On first glance that seems over engineered. I wonder what the reason is for doing it that way?

If you don't do it that way then resizing the terminal corrupts what's on screen.

Re: Claude Code daily benchmarks for degradation tracking

#330

Post by samusiam »

xnorswap wrote: post #321
isaacdl wrote: post #206

Earlier quoted context omitted.

Anywhere we can read more about what a "harness issue" means? What was the impact of it?

One thing that could be a strong degradation especially for benchmarks is they switched the default "Exit Plan" mode from: "Proceed" to "Clear Context and Proceed" It's rare you'd want to do that unless you're actually near the context window after planning. I pressed it accidentally once, and it managed to forget one of the clarifying questions it asked me because it hadn't properly written that to the plan file. If…

I disagree that this was the issue, or that it's "rare that you'd want to do that unless you're near the context window". Clearing context after writing a plan, before starting implementation of said plan, is common practice (probably standard practice) with spec driven development. If the plan is adequate, then compaction would be redundant.
Post reply on HN