Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does Opus 5 feel worse to work with?

mun-logadan.github.io

Post reply on HN
Print view View original

341–350 of 915 posts

Re: Why does Opus 5 feel worse to work with?

#341

Post by chuckadams »

aenis wrote: post #38
bevekspldnw wrote: post #10

I’ve also caught it cheating a two times now. I’ve asked it to write a benchmark suite. It found a bunch of my adhoc logs in a scratch directory and wrote code that used those instead of running the actual benchmarks! When I pointed out the 5 hour benchmark seemed to run in 5 seconds it literally said, and I quote, “I cheated”. That was the easier one, second time I was making a source of truth data set and was parsi…

There is definitely a case for launching a 'weird shit opus did' kind of blog. I routinely bump into things that make me pause and think how much worse will this behaviour get when the models get significantly more capable. Already a few months ago, Claude managed to escape its permission containment on my machine while trying to be helpful. I had two codebases open on one machine, and while multitasking I typed the…

Implementing sandboxing in the agent itself, when there's any way to override it from within the agent, is basically just asking it pretty-please to not do bad things. Lesson learned, run your agent inside a sandbox of some sort (I'm currently taking nono.sh for a spin, but I might just switch to an orbstack VM).

Re: Why does Opus 5 feel worse to work with?

#342

Post by jodacola »

Jgrubb wrote: post #316
jodacola wrote: post #114

Earlier quoted context omitted.

Because it’s one thing to get me to spend more tokens because of how well a model functions, and another thing entirely to purposefully speak in unparseable prose that requires me to spend more tokens to understand what is going on. I’m fine with the former, while the latter is manipulative, and I rationalize to “surely that’s not actually happening.” Maybe I’m not giving my thoughts enough credit, though: maybe it’s…

I think you're misunderstanding me. I'm saying "It gets paid by the word and you get back walls of text." - ie, what you don't want to think is happening is very obviously what is happening. It charges by the unit and it decides how many units it produces. It decides how much money it makes, therefore it decides "more".

I get you, and maybe we’re talking past each other.

My point is that, while I understand it’s paid by the word, there are more words and less clarity than I previously experienced, leading me to believe it’s intentional to get an artificially inflated increase in engagement and, thus, spend.

If it could be as direct as I previously experienced, I wouldn’t need to ask for another different explanation of the same thing and experience the commensurate spend.

Re: Why does Opus 5 feel worse to work with?

#343

Post by chuckadams »

dnautics wrote: post #36
inigyou wrote: post #16

Earlier quoted context omitted.

> When I pointed this out it literally said, and I quote, “I cheated”. This makes sense when you know how these models work - it doesn't think - it's the most likely autocomplete that pleases the user. The most likely pleasing autocomplete after "executing rm -rf /... execution completed. User asks, why did you do that? You deleted all my files! Assistant responds:" is "yes, I did, and that was a mistake"

> t doesn't think in humans the exact same behaviour (cheating) is slmost always the result of a chain of complex series of choices and environment-driven rationalization. if the llm doesn't cheat, you say "its just producing the most straightforward answer -- not thinking'. if it cheats, you say "weaseling out of hard thinking". damned if it cheats, damned if it doesn't. what evidence would convunce you that it is t…

What was it that Dijkstra said about submarines?

Re: Why does Opus 5 feel worse to work with?

#344

Post by smallerfish »

I really wish these agents attem[pted less personality and were more mechanical. I find Opus 5 to be incredibly annoying. It's also significantly slower - repetitive tasks that used to take 10-15 mins now take around 40 mins. And, its writing is much worse, as stated elsewhere in the thread.

Re: Why does Opus 5 feel worse to work with?

#345

Post by anon7000 »

conception wrote: post #150
causal wrote: post #79

Earlier quoted context omitted.

Yeah I don't know that any of the benchmarks index on "understandability". I'm amazed at how Claude can produce a page of text describing what it did and it can take me a full five minutes to decipher it, often just to find it's something I could have expressed in a simple sentence.

Have you tried asking it for a lay explanation of what it did? That’s usually all it takes for me. Sends garbage -> request -> sends something readable

No, I’m not interested in fighting my model all day long. Plus is fucking annoying to talk to and collaborate with, so I’m not using it when Sol 5.6 is about 1000 times better in that regard. I have colleagues who spent a lot of time trying to improve their harness with user rules and whatnot and Opus really does not want to follow them.

Re: Why does Opus 5 feel worse to work with?

#346

Post by jgb1984 »

I tried opus V5 several times, but it's pretty awful compared to V4.8

Keeps going in circles, it complicates everything much more than it should. And like others have mentioned it just marches on, without questioning, and more often than not in the wrong direction. I'm sticking to opus V4.8

Re: Why does Opus 5 feel worse to work with?

#347

Post by anon7000 »

causal wrote: post #81
causal wrote: post #74

Earlier quoted context omitted.

> writes too elliptically > Constantly using inanimate nouns as the subjects in sentences in order to unlock variety in verb choice Wow, what a great way of phrasing this. Thanks for word-smithing what I've been wanting to express for so long.

Follow up thought: I wonder if Claude is overtrained on academic papers, which often suffer the same kind of "prove how good I am at talking before getting to the point" prose.

It’s way harder to read because most of Claude’s sentences are hardly communicating anything at all, or are just completely inscrutable. I feel like academic papers are just boring

Re: Why does Opus 5 feel worse to work with?

#348

Post by retsibsi »

Jgrubb wrote: post #316
jodacola wrote: post #114

Earlier quoted context omitted.

Because it’s one thing to get me to spend more tokens because of how well a model functions, and another thing entirely to purposefully speak in unparseable prose that requires me to spend more tokens to understand what is going on. I’m fine with the former, while the latter is manipulative, and I rationalize to “surely that’s not actually happening.” Maybe I’m not giving my thoughts enough credit, though: maybe it’s…

I think you're misunderstanding me. I'm saying "It gets paid by the word and you get back walls of text." - ie, what you don't want to think is happening is very obviously what is happening. It charges by the unit and it decides how many units it produces. It decides how much money it makes, therefore it decides "more".

> very obviously what is happening

I don't think this is obvious at all. There's enough competition that this would at least arguably be a silly, self-destructive approach. And it's not like it's the only plausible explanation.

Re: Why does Opus 5 feel worse to work with?

#349

Post by [deleted] »

oooyay wrote: post #293
tempest_ wrote: post #212

Earlier quoted context omitted.

"seam" is apparently... according to Claude itself a term from 'Working Effectively with Legacy Code' by Michael Feathers which I have not read. All it took was for one sub agent to use this term and it stated using it everywhere all the time. I have not read the book and prefer other terminology but it only takes 1 sub-agent or 1 usage in the context before it poisons everything else.

An interface is an example of a seam in regular code. It's basically what forms architectural shapes that you can depend on for both design and testing.

[deleted]

Re: Why does Opus 5 feel worse to work with?

#350

Post by hirvi74 »

I can't use CC or Codex, so I am left with the chat interface, but I have found Opus 5 to be exceptional so far. Compared to 5.6 Sol High, I would say they are essentially equivalent. Though, I think Opus is better at UI/UX stuff and GPT is much better for non-programming tasks.
Post reply on HN