Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does Opus 5 feel worse to work with?

mun-logadan.github.io

Post reply on HN
Print view View original

301–310 of 915 posts

Re: Why does Opus 5 feel worse to work with?

#301

Post by demibabs »

algoth1 wrote: post #145
demibabs wrote: post #44

Earlier quoted context omitted.

I wouldn’t exactly put it like that. It’s moreso the model sometimes outputting non-optimal tokens in a way that’s detectable if you know the algorithm. It seems possible for that to make the response “drift” far from what it would’ve been, because it’s constant entropy that adds up after time. (However, according to Anthropic and Google, it doesn’t really impact the quality of responses. I find that a bit hard to be…

Yeah, it’s hard to believe, specially when you are coding and there’s only one best way to do things, unless it plays with variable naming, or comments

Well the algorithm only increases the chances of certain words being chosen/not chosen, rather than guaranteeing it. If there’s a clear answer then that nudge won’t do anything.

If the model’s most recent output is “for (let i = 0; ”, the likelihood of the next token being “i” is probably millions of times greater than any other possible token. Thus even if “i” is on the red list and has its likelihood decreased, it’s not going to suddenly choose another word.

Put another way, on low-entropy tasks like coding, this style of fingerprinting is less effective and needs bigger sample sizes to be recognizable.

That said, even small changes can dramatically affect output quality, which is why I’m still a skeptic.

Re: Why does Opus 5 feel worse to work with?

#303

Post by saaaaaam »

Mtinie wrote: post #187
saaaaaam wrote: post #153

Earlier quoted context omitted.

> Of course, it also has no concept of what context the user has and assumes that it must be the same it holds in its memory, which creates this “I didn’t know that you didn’t know” type of communication. Yes, this is a repeated problem for me. It will drop something in as though we have discussed it before and when I say “hold on, what is this” it realises its error - though on more than one occasion has started to…

I notice the models with reasoning can conflate “internal” (or subagent) discussions with external (i.e. me). So it is accurately indicating “I’ve had this discussion before” but incorrectly asserting who it was with. My understanding of how “thinking”works is limited though, and given the reduced visibility into the thinking traces, it is harder to tell if this is actually happening or if these are imaginary discuss…

Oh, that's interesting - because that's absolutely what's happening in my experience.

If I look at the thinking (which seems to have become unavailable in Opus 5 a lot of the time, but was present - and often useful - in 4.8/4.6) you're right - it's having the discussion with itself, and seems unable to distinguish that discussion from discussions with me. BUT it also seems to be related to the length of the chat - this seems far more likely to happen in a longer chat.

I don't understand why they have removed visibility into thinking - I found it very useful, not only for spotting things like this, but also because in more complex discussions it would often mention (useful) things in its train of thought that it dropped from its response - but if I said "when you were thinking, you mentioned this" it would then expand on that point. Taking that away is another thing that has negatively impacted the value I get from Opus 5.0 versus earlier models.

Re: Why does Opus 5 feel worse to work with?

#304

Post by anentropic »

> stop and ask questions if my intent was unclear,

> don't make assumptions without checking,

> and don't reinterpret or update my plans without asking.

these aren't at all the problems I have with it

I have found it good at asking questions, to the extent I rarely use 'plan mode' any more

but often it's hard to understand what it's asking me, it's like the question framing has been pulled from the middle of its own reasoning stream, references aren't anchored or restated, often I have to prompt it to ask again but "clearly and concisely, for humans"

Re: Why does Opus 5 feel worse to work with?

#305

Post by sixdimensional »

repeekad wrote: post #267
sixdimensional wrote: post #229

Anthropic, if you're listening - by the time this crops up on Reddit, the front page of HN, etc.... you should be expecting calls from CEOs of major corporations next threatening to abandon ship... We've seen this pattern before several times.. I hope they are listening and address this publicly. I'm not sure what is going on, some users report it works fine or great, others report the degradation. I've experienced b…

> owe it to society In America? lol if only, only a law would get them to act for that reason, maybe not even that these days..

Or, maybe competition.. but your point is taken.

I like to hope that those in positions of power do have a sense of morality though too.. but their worldview is quite different than an ordinary citizen.

Re: Why does Opus 5 feel worse to work with?

#306

Post by empath75 »

nathanyz wrote: post #210
slaser79 wrote: post #188

A lot of the issues have been already noted here..Two "regressions" for me: 1. Communication ability. It basically now speaks almost in riddles I am asking OPUS 5 for tldrs all the time now (should skillify it now!) 2. Overengineers for edge cases. I get it. With all the benchmarking and RLing, but now tasks that would have been completed relatively quick take much longer as it overengineers all the edge cases, and s…

Your #2 is spot on. I have in fact said the same exact words to Opus about missing the forest for the trees. I have it work on some code for an inhouse ClaudeCode plugin, and it starts coding as if it will be attacked by hackers who will try all sorts of variations to break it. I can appreciate that in cases of software that is public facing or accessible, but for a simple helper plugin it is overkill. It will even a…

Yeah, trying to get it to add features is sometimes _impossible_. It'll circle around and around creating all kinds of preconditions that you never asked for as excuses for why it can't do the really simple thing you asked for.

It'll also find some minor security problem and drop everything on the floor with URGENT without me asking it to.

Re: Why does Opus 5 feel worse to work with?

#307

Post by chuckadams »

oooyay wrote: post #293
tempest_ wrote: post #212

Earlier quoted context omitted.

"seam" is apparently... according to Claude itself a term from 'Working Effectively with Legacy Code' by Michael Feathers which I have not read. All it took was for one sub agent to use this term and it stated using it everywhere all the time. I have not read the book and prefer other terminology but it only takes 1 sub-agent or 1 usage in the context before it poisons everything else.

An interface is an example of a seam in regular code. It's basically what forms architectural shapes that you can depend on for both design and testing.

It's a fairly good concise term ... load-bearing, even. /ducks

But even then, I think "boundary" was the more common term before some LLM decided it really liked "seam" instead.

Re: Why does Opus 5 feel worse to work with?

#308

Post by ACS_Solver »

droserasprout wrote: post #182
ACS_Solver wrote: post #100

Earlier quoted context omitted.

I also suspect comments are very much tied to how Claude reasons because not only are they bad comments, I can't get rid of them. Commenting is the one area in which I've been unable to get Claude to respect any rules. It can follow code conventions I prefer, it can do other things, but it can't keep the comment volume down. My CLAUDE.md has rules about not including any redundant comments in the code that are obviou…

> Commenting is the one area in which I've been unable to get Claude to respect any rules. Exactly my experience! Since the release of Opus 5, no amount of instructions helps. In CLAUDE.md, in a separate file, in memory, as brief bullets, as long detailed guides, with reasoning from medium to max — nothing. Even worse, recently, after getting another opus in a tiny bugfix session, I prompted directly, "drop the comme…

> Even worse, recently, after getting another opus in a tiny bugfix session, I prompted directly, "drop the comments from the current code changes" — Claude instead just slightly trimmed them. I couldn't believe my eyes.

For me, Claude knows how I want the comments due to all the memories and CLAUDE.md, so funnily it's now enough with even a brief groan from me like "Come on, the comments" and then Claude goes through its recent additions and fixes comments quite well per my long-term instructions. But only ever during an extra pass that I initiate, never during the initial writing of the code.

Re: Why does Opus 5 feel worse to work with?

#309

Post by mattmaynes »

I thought I was going crazy. I was late to upgrade from 4.8 to 5 and I already want to go back. I’ve noticed that in addition to taking liberties with my instructions, it is also less capable at debugging its own issues. In trying to fix a problem in my CI pipeline, it went chasing some “quadratic race condition” it claimed. It turned out that it had added a loop in a test at the wrong level and it spent an hour chasing it down before I realized what was going on. I pointed 4.8 at the same problem and it solved it in 2 minutes.

I certainly agree with the original post. It feels like the model has been highly benchmark tailored and it is now worse at solving problems that fall outside of the standard patterns.

Re: Why does Opus 5 feel worse to work with?

#310

Post by saaaaaam »

purplepatrick wrote: post #236
Mtinie wrote: post #187

Earlier quoted context omitted.

I notice the models with reasoning can conflate “internal” (or subagent) discussions with external (i.e. me). So it is accurately indicating “I’ve had this discussion before” but incorrectly asserting who it was with. My understanding of how “thinking”works is limited though, and given the reduced visibility into the thinking traces, it is harder to tell if this is actually happening or if these are imaginary discuss…

Yeah, basically everything that becomes context in a session will bias perception and communication style -- subagents, plan lingo, prompt lingo, etc. And then if you write a plan with the comms context having been biased, the lingo will creep into the plan, and from the plan into the code and code comments. And from there, bad lingo will go on multiplying like rabbits... I usually think of it in terms of having a "g…

> I usually think of it in terms of having a "good" or "bad" session. In a bad session, there is a harmful bias that you can only get rid of through a new session. For example, if you exposed too much context about, say, a variable that features prominently in a doc. The entire session will be anchoring on the importance of that variable. Or if you introduced the notion of CC having to ask for permission for stuff you will have a hard time getting it to "think on its feet" or propose an effective solution (you have made CC so insecure that it now relies on you even for little things that wouldn't normally require your input). In some cases (let's say you have important context in that session) you can overcome this by upping the reasoning level or switching to Fable, but usually a new session is the way to go.

I find this very interesting, particularly your points about "made CC so insecure". I know that we have a tendency to anthropomorphise around these tools, but I have definitely noticed instances where Claude becomes quite hysterical about things - and if you look in the thinking output, it's often after I've pushed back on something, or told it it is going in the wrong direction. It spends a lot of time in agonised second-guessing of itself, going round in circles, before outputting a cringeing hand-wringing response. It's very strange.

Good tip on upping the reasoning level - I've not tried this. I have tried switching to Fable though, which does help. But it obviously very hungry, particularly in longer chats because it presumably needs to remind itself of everything that has occurred so far in the chat.

The point you make about tools that pretend to give Claude "a brain" or "remember" things is also interesting - I find the "memory" feature in Claude so destructive to good outputs that when I'm using the chat interface I am very strict about using Projects, and usually turn off the project memory, or make efforts to manage the project memory and review and delete things that are skewing the outputs.

Post reply on HN