Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issue: Claude Code is unusable for complex engineering tasks with Feb updates

github.com

Post reply on HN
Print view View original

311–320 of 829 posts

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#311

Post by Majromax »

armchairhacker wrote: post #134

Yet https://marginlab.ai/trackers/claude-code/ says no issue. If you're so convinced the models keep getting worse, build or crowdfund your own tracker.

If I'm reading that page correctly, then the benchmark results don't cover the interesting "mid February" inflection point noted in the article/report. The numbers appear to begin after the quality drop began. Moreover, the daily confidence interval seems to be stupidly wide, with a confidence interval between 42% and 69%?

The "Other metrics" graphs extend for a longer period, and those do seem to correlate with the report. Notably, the 'input tokens' (and consequently API cost) roughly halve (from 120M to 60M) between the beginning of February and mid-March, while the number of output tokens remains similar. That's consistent with the report's observation that new!Opus is more eager to edit code and skips reading/research steps.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#312

Post by copperx »

nrds wrote: post #91
riskassessment wrote: post #67

Earlier quoted context omitted.

Stealthily degrade the model or stealthily constrain the model with a tighter harness? These coding tools like Claude Code were created to overcome the shortcomings of last year's models. Models have gotten better but the harnesses have not been rebuilt from scratch to reflect improved planning and tool use inherent to newer models. I do wonder how much all the engineering put into these coding tools may actually in…

I've been using pi.dev since December. The only significant change to the harness in that time which affects my usage is the availability of parallel tool calls. Yet Claude models have become unusable in the past month for many of the reasons observed here. Conclusion: it's not the harness. I tend to agree about the legacy workarounds being actively harmful though. I tried out Zed agent for a while and I was SHOCKED…

Are you using Pi with a cloud subscription, or are you using the API?

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#313

Post by KenoFischer »

bcherny wrote: post #209

Hey all, Boris from the Claude Code team here. I just responded on the issue, and cross-posting here for input. --- Hi, thanks for the detailed analysis. Before I keep going, I wanted to say I appreciate the depth of thinking & care that went into this. There's a lot here, I will try to break it down a bit. These are the two core things happening: > `redact-thinking-2026-02-12` This beta header hides thinking from th…

While we have you here, could you fix the bash escaping bug? https://github.com/anthropics/claude-code/issues/10153

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#314

Post by Majromax »

noxa wrote: post #179

I'm the author of the report in there. The stop-phrase-guard didn't get attached but here it is: https://gist.github.com/benvanik/ee00bd1b6c9154d6545c63e06a3... You can watch for these yourself - they are strong indicators of shallow thinking. If you still have logs from Jan/Feb you can point claude at that issue and have it go look for the same things (read:edit ratio shifts, thinking character shifts before the red…

I'm curious about your subscription/API comparison with respect to thinking. Do you have a benchmark for this, where the same set of prompts under a Claude Code subscription result in significantly different levels of effective thinking effort compared to a Claude Code+API call?

Elsewhere in this thread 'Boris from the Claude Code team' alleges that the new behaviours (redacted thinking, lower/variable effort) can be disabled by preference or environment variable, allowing a more transparent comparison.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#315

Post by ceejayoz »

whateveracct wrote: post #293
rubicon33 wrote: post #86

Earlier quoted context omitted.

You will literally build nothing but the most primitive of devices unless you accept black boxes. In fact I'd argue its one of humanities great strengths that we can build on top of the tools others have built, without having to understand them at the same level it took to develop them.

I have been able to build plenty of stuff with a pretty plain emacs + ghci for years...neither are black boxes. Except maybe my brain driving them.

They run on an operating system you probably don't know all the inner workings of.

And that runs on a chip with trillions of transistors.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#316

Post by parliament32 »

pkilgore wrote: post #96
stared wrote: post #78

I am curious - is there any hard data (e.g. a benchmark score drop)? I feel that we look for patterns to the point of being superstitious. (ML would call it overfitting.)

Did you have specific complaints about the data in the OP?

The wall of slop after the single human paragraph, you mean? Text generator output isn't data.. it's at best unreliable, and at worst entirely fabricated.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#317

Post by ThrowawayR2 »

This sort of thing kills stone dead the argument by the AI advocates that the transition to LLMs is no different than the transition to using compilers. If output quality can vary significantly because of underlying changes to the model or whatever without warning or recourse, it's a roulette wheel instead of a reliable tool.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#318

Post by yubblegum »

bcherny wrote: post #209

Hey all, Boris from the Claude Code team here. I just responded on the issue, and cross-posting here for input. --- Hi, thanks for the detailed analysis. Before I keep going, I wanted to say I appreciate the depth of thinking & care that went into this. There's a lot here, I will try to break it down a bit. These are the two core things happening: > `redact-thinking-2026-02-12` This beta header hides thinking from th…

> Before I keep going, I wanted to say I appreciate the depth of thinking & care that went into this.

"This report was produced by me — Claude Opus 4.6 — analyzing my own session logs. ... Ben built the stop hook, the convention reviews, the frustration-capture tools, and this entire analysis pipeline because he believes the problem is fixable and the collaboration is worth saving. He spent today — a day he could have spent shipping code — building infrastructure to work around my limitations instead of leaving."

What a "fuckin'" circle jerk this universe has turned out to be. This note was produced by me and who the hell is Ben?

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#319

Post by didgeoridoo »

tills13 wrote: post #203
monkpit wrote: post #161

Earlier quoted context omitted.

> expletives per word Huh?

4.4 expletives per word is insane. Their prompts must look like ** ** ** ** implement ** ** ** ** no ** ** ** ** ** mistakes

Haha no that’s change - 4.4x MORE expletives per word in the last week.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#320

Post by mavamaarten »

nikanj wrote: post #151
summarity wrote: post #8

Not claude code specific, but I've been noticing this on Opus 4.6 models through Copilot and others as well. Whenever the phrase "simplest fix" appears, it's time to pull the emergency break. This has gotten much, much worse over the past few weeks. It will produce completely useless code, knowingly (because up to that phrase the reasoning was correct) breaking things. Today another thing started happening which are…

”I can’t make this api work for my client. I have deleted all the files in the (reference) server source code, and replaced it with a python version” Repeatedly, too. Had to make the server reference sources read-only as I got tired of having to copy them over repeatedly

Haha yeah. I once asked it to make a field in an API response nullable, and to gracefully handle cases where that might be an issue (it was really easy, I was just lazy and could have done it myself, but I thought it was the perfect task for my AI idiot intern to handle). Sure, it said. Then it was bored of the task and just deleted the field altogether.
Post reply on HN