Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issue: Claude Code is unusable for complex engineering tasks with Feb updates

github.com

Post reply on HN
Print view View original

781–790 of 829 posts

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#781

Post by tomaskafka »

bcherny wrote: post #461
JamesSwift wrote: post #397

Earlier quoted context omitted.

a9284923-141a-434a-bfbb-52de7329861d d48d5a68-82cd-4988-b95c-c8c034003cd0 5c236e02-16ea-42b1-b935-3a6a768e3655 22e09356-08ce-4b2c-a8fd-596d818b1e8a 4cb894f7-c3ed-4b8d-86c6-0242200ea333 Amusingly (not really), this is me trying to get sessions to resume to then get feedback ids and it being an absolute chore to get it to give me the commands to resume these conversations but it keeps messing things up: cf764035-0a1d-4…

Thanks for the feedback IDs — read all 5 transcripts. On the model behavior: your sessions were sending effort=high on every request (confirmed in telemetry), so this isn't the effort default. The data points at adaptive thinking under-allocating reasoning on certain turns — the specific turns where it fabricated (stripe API version, git SHA suffix, apt package list) had zero reasoning emitted, while the turns with d…

My guess is there isn't enough hardware, so Anthropic is trying to limit how much soup the buffet serve, did I guess right? And I would absolutely bet the enterprise accounts with millions in spend get priority, while the retail will be first to get throttled.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#783

Post by pcjones1 »

ghusbands wrote: post #731
pcjones1 wrote: post #703

Earlier quoted context omitted.

Have you set effort to high or max?

Even with high effort, the adaptive thinking can just choose no thinking. See bcherny's post they were replying to: https://news.ycombinator.com/item?id=47668520

Yeah I know but you can disable it as we saw

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#784

Post by olejorgenb »

kouteiheika wrote: post #728
Wowfunhappy wrote: post #509

Earlier quoted context omitted.

> Under the hood, by setting this header we avoid needing thinking summaries, which reduces latency. You can opt out of it with `showThinkingSummaries: true` in your settings.json (see [docs]( https://code.claude.com/docs/en/settings#available-settings )). Can I just see the actual thinking (not summarized) so that I can see the actual thinking without a latency cost? I do really need to see the thinking in some form…

> Can I just see the actual thinking (not summarized) so that I can see the actual thinking without a latency cost? You can't, and Anthropic will never allow it since it allows others to more easily distill Claude (i.e. "distillation attacks"[1] in Anthropic-speak, even though Athropic is doing essentially exactly the same thing[2]; rules for thee but not for me). [1] -- https://www.anthropic.com/news/detecting-and-p…

So this means I can not resume a session older than 30 days properly?

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#785

Post by richardjennings »

thesz wrote: post #780
richardjennings wrote: post #600

Earlier quoted context omitted.

I think over-thinking is only solved by thinking more, not less. This is only viable once some intelligence threshold is reached, which I think Anthropic has borderline achieved.

> I think over-thinking is only solved by thinking more, not less. Despite "thinking" tokens being determined by the preceding tokens, they still are taken from some probability distribution, just a complex one. This means that at each token selection step there is a probability P_e of an error, of selecting a wrong token. These errors compound exponentially: the probability of not selecting wrong token for N steps i…

> The shorter "thinking" is, the less is the probability of it going astray

As long as the error introduced by more steps is less than the compounding error of sub-optimal token sampling, I would expect a better result.

I think your choice of "wrong" is extreme, suggesting such a token can catastrophically spoil the result. The modern reality is more that the model is able to recover.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#786

Post by hansmayer »

SAI_Peregrinus wrote: post #734
hansmayer wrote: post #653

Earlier quoted context omitted.

I would laugh so hard at this, if your attempt at comparison was not so tragic. Bash and other shells are deterministic. Want to set it just for one user ? - use ~/.bashrc . Set it for all users on the system? use /etc/profile.d/ . Want it just temporary for this session? You got it, environment variables. And it is going to work like that every single time. It is deterministic you see.

The non-determinisim in the LLM systems isn't because of the different config uses, that works much like shell configs. The non-determinism is inherent in LLM operations.

Exactly my point here...

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#787

Post by robeym »

jnfr wrote: post #747
robeym wrote: post #707

Earlier quoted context omitted.

This is confusing. ULTRATHINK is a step below /effort max? ULTRATHINK triggers high effort. /effort max is above high. Calling it ULTRATHINK sounds like it would be the highest mode. If someone has max set and types ULTRATHINK, they're lowering their effort for that turn. For anyone reading this trying to fix the quality issues, here's what I landed on in ~/.claude/settings.json: { "env": { "CLAUDE_CODE_EFFORT_LEVEL"…

Thanks for sharing. Have you experienced noticeable impact to your usage rate?

Nothing super noticeable. I've reached 35% in sessions on the 20x plan. Before these changes, 25-30% was pretty normal. I think these changes are best for people who are just past the 5x usage plan, but might be harder to manage if you already have to throttle usage to stay under limits.

I'd still recommend turning off sub agents entirely because it doesn't seem you can control them with /effort and I always find the output to be better with agents off.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#788

Post by sigbottle »

rimliu wrote: post #671
sigbottle wrote: post #193

Earlier quoted context omitted.

I think the terminology is just dogshit in this area. LLMs are great semantic searchers and can reason decently well - I'm using them to self teach a lot of fields. But I inevitably reach a point where I come up with some new thoughts and it's not capable of keeping up and I start going to what real people are saying right now, today, and trust the LLM less and instead go to primary sources and real people. But I wou…

do they reason? Where was a video by AI researcher, that showed, that they do not reason but actually come with the result first and then try to invent "reasoning" to match it.

I mean humans do that too, and I don't think it's very unjustified. The "we deduce from a deep base premise P down a chain of inferences" picture is extremely incomplete and has been challenged all over the place - by normal people, by analytic and continental philosophers, by science itself, etc.

Not trying to say that LLM's are equivalent to humans but that the concept of reasoning is undefined.

And the fact that their performance does increase when using test-time compute is empirical evidence that they're doing something that increases their performance on tasks that we consider would require reasoning. As to what that is, we don't know.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#789

Post by matheusmoreira »

gck1 wrote: post #597
faitswulff wrote: post #513

Earlier quoted context omitted.

Anthropic's position is that thinking tokens aren't actually faithful to the internal logic that the LLM is using, which may be one reason why they started to exclude them: https://www.anthropic.com/research/reasoning-models-dont-say...

That probably matters for some scenarios, but I have yet to find one where thinking tokens didn't hint at the root cause of the failure. All of my unsupervised worker agents have sidecars that inject messages when thinking tokens match some heuristics. For example, any time opus says "pragmatic", its instant Esc Esc > "Pragmatic fix is always wrong, do the Correct fix", also whenever "pre-existing issue" appears (it'…

> also whenever "pre-existing issue" appears (it's never pre-existing)

I dunno... There were some pre-existing issues in my projects. Claude ran into them and correctly classified as pre-existing. It's definitely a problem if Claude breaks tests then claims the issue was pre-existing, but is that really what's happening?

I agree with the correctness issue.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#790

Post by rachel_rig »

causal wrote: post #215
onlyrealcuzzo wrote: post #115

Earlier quoted context omitted.

> Whenever the phrase "simplest fix" appears, it's time to pull the emergency break. Second! In CLAUDE.md, I have a full section NOT to ever do this, and how to ACTUALLY fix something. This has helped enormously.

I switched from Cursor to Claude because the limits are so much higher but I see Anthropic playing a lot more games to limit token use

[flagged]
Post reply on HN