Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does Opus 5 feel worse to work with?

mun-logadan.github.io

Post reply on HN
Print view View original

771–780 of 916 posts

Re: Why does Opus 5 feel worse to work with?

#771

Post by senderista »

Retr0id wrote: post #84
8cvor6j844qw_d6 wrote: post #73

Earlier quoted context omitted.

It's bad enough that I've seen dedicated skills to do comment hygiene scrubbing and consolidation.

I've tried telling it to "fix" comments with varying degrees of specificity and in my experience it just... fundamentally doesn't get it. Presumably using a different model for it would help. My theory is that Claude's learned approach to comments is to treat them as a sort of persistent in-band thinking trace, or a "memory" tied to an in-code location, which is a little at odds with the way humans use comments (huma…

I have had some luck telling Sol to concisely rephrase Opus 5’s comments.

Re: Why does Opus 5 feel worse to work with?

#774

Post by ngruhn »

modo_ wrote: post #429
ngruhn wrote: post #424

Earlier quoted context omitted.

I'm switching to GPT because of this. The prose is so much more legible. The only reason I keep using Claude Code is because the harness is the best IMO.

Your point on the harness is interesting. How do you distinguish characteristics of the model from characteristics of the harness? In the early days I feel it was more apparent. You would frequently see the model making failed tool calls etc.. but now that feels so rare. I'm not confident I can perceive whatever shortcomings of the harness remain.

I'm not talking about model performance. I just mean the UX of Claude Code. I'm trying to use pi but there are so many paper cuts. Of course you can configure everything but that's a ton of work. Claude Code has pretty good defaults.

Re: Why does Opus 5 feel worse to work with?

#776

Post by weitendorf »

fidotron wrote: post #503
D13Fd wrote: post #120

I’ve been doing some heavy work on a personal project lately. I burned through the limits on Claude, the plus a few hundred dollars in credits, and ultimately decided to move to an OpenAI account just so I can keep going. I was surprised to find that OpenAI Sol is much much nicer to work with than Opus 5 or Fable at the moment. Especially on Opus 5, the way it communicates is just exhausting. It keeps “being honest”…

It's so obnoxious it has to be deliberate, like a colleague from hell. There's a constant strand from the AI safety brigade that "people get used to sycophantic LLMs which give them unrealistic expectations of human interaction" so Anthropic are overcompensating by making their models verging on antagonistic to deal with, so that we stay appreciative of our human brethren or something. They seem to have forgotten the…

I think it's much simpler than that, they are indeed trying to counter against sycophancy and models lying (the decision whether or not to lie to the human user, take shortcuts, etc. comes up in their thinking traces pretty commonly IIUC) by rewarding them for giving good-and-bad feedback, fessing up to things, pointing out potential issues, etc.

The problem is just that they are rewarding the behavior shallowly, ie rewarding the appearance of honesty or neutral replies, being highly detailed/thorough, even where it doesn't make sense to do so.

I think this is partly due to a reliance on LLM-as-judge training runs/synthetic data during RL where they're having a model which itself doesn't epistemically understand when this behavior is necessary or valuable influence the feedback provided to the model being trained. And that's mostly a problem of scale/volume and the desire to have a tight feedback loop rather than a safety issue IMO. They just generate an absurd amount of traces during training and the only way to really evaluate/rank/steer them at the scale they're generated is through other models, and combined with some kind of honesty/truthfulness/non-sycophancy eval that isn't robust enough to prevent mode collapse, you get this.

Re: Why does Opus 5 feel worse to work with?

#777

Post by cebert »

galaxyLogic wrote: post #535
saaaaaam wrote: post #303

Earlier quoted context omitted.

Oh, that's interesting - because that's absolutely what's happening in my experience. If I look at the thinking (which seems to have become unavailable in Opus 5 a lot of the time, but was present - and often useful - in 4.8/4.6) you're right - it's having the discussion with itself, and seems unable to distinguish that discussion from discussions with me. BUT it also seems to be related to the length of the chat - t…

Interesting. So if the LLM is having a discussion with itself, am I paying for the tokens it uses for that?

> So if the LLM is having a discussion with itself, am I paying for the tokens it uses for that?

You absolutely pay for them. This is why changing effort/reasoning levels have such a significant impact on session cost.

Re: Why does Opus 5 feel worse to work with?

#778

Post by D13Fd »

sejje wrote: post #389
D13Fd wrote: post #120

I’ve been doing some heavy work on a personal project lately. I burned through the limits on Claude, the plus a few hundred dollars in credits, and ultimately decided to move to an OpenAI account just so I can keep going. I was surprised to find that OpenAI Sol is much much nicer to work with than Opus 5 or Fable at the moment. Especially on Opus 5, the way it communicates is just exhausting. It keeps “being honest”…

> I was surprised to find that OpenAI Sol is much much nicer to work with than Opus 5 or Fable at the moment. Why were you surprised?

Because I felt the opposite a while back when I switch from OpenAI->Anthropic, and because I understood that the two companies were basically at parity right now on their frontier models.

Re: Why does Opus 5 feel worse to work with?

#779

Post by indrex »

barrkel wrote: post #12

The single biggest annoyance with Opus 5 is that it writes too elliptically. Sentences that orbit a point, then jump to it like it's a revealed insight. Unnecessarily abstract phraseology. Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice, especially when it helps construct a sentence where the real action can 'land' like a surprise at the end. It is definitely mo…

I had to setup hooks just to catch the and ban them in the CLIs. Most of what Claude writes is simply un-readable.

Re: Why does Opus 5 feel worse to work with?

#780

Post by valleyer »

eli_gottlieb wrote: post #691
purplepatrick wrote: post #63

Earlier quoted context omitted.

Agreed. CC’s comms capabilities have decreased gradually since 4.6, and it’s a real challenge. I think the issue is that what works well for code (succinctness) doesn’t work well in prosaic English. CC’s communication violates almost every grammatical rule that’s tested on, say, the SAT. And yet I’m sure if you had Claude take the verbal section of the exam it would ace it. Biggest issues: dense sentences, constant m…

> and it’s a real challenge. What model did you use to write this?

Yeah, "X, and it's Y" is a common trope I see.
Post reply on HN