Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Issue: Claude Code is unusable for complex engineering tasks with Feb updates

github.com

Post reply on HN
Print view View original

271–280 of 829 posts

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#271

Post by ai_slop_hater »

koverstreet wrote: post #261
tatrions wrote: post #247

Earlier quoted context omitted.

[flagged]

Technically speaking, models inherently do this - CoT is just output tokens that aren't included in the final response because they're enclosed in tags, and it's the model that decides when to close the tag. You can add a bias to make it more or less likely for a model to generate a particular token, and that's how budgets work, but it's always going to be better in the long run to let the model make that decision en…

> You can add a bias to make it more or less likely for a model to generate a particular token, and that's how budgets work

Do you have a source for this? I am interested in learning more about how this works.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#273

Post by NooneAtAll3 »

riskassessment wrote: post #67
matheusmoreira wrote: post #33

That analysis is pretty brutal. It's very disconcerting that they can sell access to a high quality model then just stealthily degrade it over time, effectively pulling the rug from under their customers.

Stealthily degrade the model or stealthily constrain the model with a tighter harness? These coding tools like Claude Code were created to overcome the shortcomings of last year's models. Models have gotten better but the harnesses have not been rebuilt from scratch to reflect improved planning and tool use inherent to newer models. I do wonder how much all the engineering put into these coding tools may actually in…

I feel like "feature/model freeze" may be justified

just call it something like "[month][year]edition" and work on next release

users spend effort arriving to narrow peak of performace, but every change keeps moving the peak sideways

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#274

Post by ctoth »

quietsegfault wrote: post #260
ctoth wrote: post #256

Earlier quoted context omitted.

[flagged]

I’m not sure being confrontational like this really helps your case. There are real people responding, and even if you’re frustrated it doesn’t pay off to take that frustration out on the people willing to help.

Fair point on tone. It's a bit of a bind isn't it? When you come with a well-researched issue as OP did, you get this bland corporate nonsense "don't believe your lyin' eyes, we didn't change anything major, you can fix it in settings."

How should you actually communicate in such a way that you are actually heard when this is the default wall you hit?

The author is in this thread saying every suggested setting is already maxed. The response is "try these settings." What's the productive version of pointing out that the answer doesn't address the evidence? Genuine question. I linked my repo because it's the most concrete example I have.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#275

Post by Retr0id »

malfist wrote: post #270
quietsegfault wrote: post #260

Earlier quoted context omitted.

I’m not sure being confrontational like this really helps your case. There are real people responding, and even if you’re frustrated it doesn’t pay off to take that frustration out on the people willing to help.

Is somebody saying "you're holding it wrong" a "people willing to help"?

[flagged]

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#277

Post by garfij »

jwr wrote: post #204

I wish they had a "and we won't screw you in two weeks" plan at, say, 5x the price. It's worth it for my business, I'd pay it. Should I switch back to API pricing? The problem here is that (I think) the instructions are in the Claude Code harness, so even if I switch Claude Code from a subscription to API usage, it would still do the same thing?

FWIW I've only ever been on the API based plan at work and we never seem to run into the majority of the problems people seem to be very vocal about. Outages still affect us, and we do have the intermittent voodoo feeling of "Claude seems stupider today", but nothing persistent.

Of course it's a stupid amount of money sometimes, but I generally feel like we get what we're paying for.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#278

Post by [deleted] »

toenail wrote: post #59
phillipcarter wrote: post #11

Maybe it's because I spend a lot of time breaking up tasks beforehand to be highly specific and narrow, but I really don't run into issues like this at all. A trivial example: whenever CC suggests doing more than one thing in a planning mode, just have it focus on each task and subtask separately, bounding each one by a commit. Each commit is a push/deploy as well, leading to a shitload of pushes and deployments, but…

I thought everybody does this.. having a model create anything that isn't highly focused only leads to technical debt. I have used models to create complex software, but I do architecture and code reviews, and they are very necessary.

[deleted]

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#279

Post by gchamonlive »

Retr0id wrote: post #252
gchamonlive wrote: post #239

Earlier quoted context omitted.

How should one conduct such a rigourously reproducible experiment when LLMs by nature aren't deterministic and when you don't have access to the model you are comparing to from months ago?

Something like this: https://marginlab.ai/trackers/claude-code/ (see methodology section)

Kudos for the methodology. The only question I can come up with is that if the benchmarks are representative of daily use.

Anecdotal or not, we see enough reports popping up to at least elicit some suspion as to service degradation which isn't shown in the charts. Hypothesis is that maybe the degradation experienced by users, assuming there is merit in the anecdotes, isn't picked up by the kind of tracking strategy used.

Re: Issue: Claude Code is unusable for complex engineering tasks with Feb updates

#280

Post by root_axis »

jwr wrote: post #196
efficax wrote: post #191

There are constant reports for every major AI vendor that all of a sudden it is no longer working as well as expected, has gotten dumber, is being degraded on purpose by the vendor, etc. Isn't the more economical explanation that these models were never as impressive as you first thought they were, hallucinate often, break down in unexpected ways depending on context, and simply cannot handle large and complex engine…

That's one of the possible explanations, but I think too many people are seeing the same symptoms (and some actually measured them). An "economical explanation" is actually that Anthropic subscriptions are heavily subsidized and after a while they realized that they need to make Claude be more stingy with thinking tokens. So they modified the instructions and this is the result.

> but I think too many people are seeing the same symptoms (and some actually measured them).

Or too many people are slurping up anecdotes from the same watering hole that confirms their opinions. Outside of academic papers, I don't think I've ever seen an example of "measuring" output that couldn't also be explained by stochastic variability.

Post reply on HN