Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does Opus 5 feel worse to work with?

mun-logadan.github.io

Post reply on HN
Print view View original

721–730 of 916 posts

Re: Why does Opus 5 feel worse to work with?

#721

Post by pjm331 »

zmmmmm wrote: post #674

I'm with the author and others in this comment thread, speculating that effectively the balance has tipped to where humans are no longer the target audience of post training - other agents are. Whether it's through the reasoning / CoT, or whether it's in handing off to subagents etc, the focus has moved to agents communicating in "agent-speak" to themselves or other agents. And human niceties are just kind of, noise…

or these models move to background agents and we need new ones for the humans to talk to

Re: Why does Opus 5 feel worse to work with?

#722

Post by Exoristos »

peddling-brink wrote: post #694
sshine wrote: post #600

Earlier quoted context omitted.

> if I was talking to a person who constantly used a phrase they liked I would notice it and it is possible I might get irritated by it There are sociological reasons why this happens less with humans: 1. You cycle your dumb repetitive jokes with everyone you meet, so nobody hears it twice 2. Those who know you well will notice when you're just repeating ("dad jokes") 3. As a person's idiosyncrasies are beginning to…

This is conjecture, but why couldn't they hire the writing equivalent of a voice actor? Honestly I don't think it would take much. Doing it ethically would take a little more money, but not much, for them. Hire a prolific author with the writing equivalent of the "midwestern accent". I have a terrible writing accent, so it couldn't be me, but these people are out there. Pay them a bunch of dollars to ingest their ent…

Boy do you underestimate everyday human abilities.

Re: Why does Opus 5 feel worse to work with?

#723

Post by elgertam »

inigyou wrote: post #710
elgertam wrote: post #127

Earlier quoted context omitted.

If I could give it a novel task outside of its explicit training and see it actually improve just through accreting context, I'd be convinced it was thinking. The opposite happens in practice. I test new models with two tasks: iteratively generating SVGs based on a text description with rendered rasters for feedback; and generating "Before and After" clues like on Jeopardy, where the response has two overlapping phra…

Now that you've spoiled it, models will be trained on these benchmarks, especially the second one that's very specific.

The second one has caused hours of entertainment over the past year or so. My kids find the LLM's failed clues and profuse apologies for getting these wrong hilarious, so it's become a family activity with me performing dramatic readings of the chat transcript with them. The LLM's apologies also seem to get more exaggerated as the context increases and the LLM seems to get more deranged.

I'd prefer the models to get better at SVG. I really hate working with the rasters that diffusion models generate, but the vector outputs are just really bad even when tokenizable like SVG. I've done some experimentation with trying to make these work better with some newer techniques with some success. But I also think the SVG Paths mini-language may be a bit too concise and unforgiving for LLMs to consistently get them right without specialized training.

Re: Why does Opus 5 feel worse to work with?

#724

Post by lubujackson »

zmmmmm wrote: post #674

I'm with the author and others in this comment thread, speculating that effectively the balance has tipped to where humans are no longer the target audience of post training - other agents are. Whether it's through the reasoning / CoT, or whether it's in handing off to subagents etc, the focus has moved to agents communicating in "agent-speak" to themselves or other agents. And human niceties are just kind of, noise…

Every round of models (plus all the secret tweaks) require new strategies to stay afloat as a human. My new tactic for Fable and Opus is to give them a line limit, both during planning and code creation. It os amazing how well that works for keeping them on task and avoiding premature optimization, pointless tests or any of those "robustness" ideas that are not planned or asked for.

Re: Why does Opus 5 feel worse to work with?

#725

Post by claiir »

adamcharnock wrote: post #232

My latest trick (literally from yesterday) is to just ask it to write according to ISO 24495-1, the standard for plain language: > [This standard is] for anybody who creates or helps create documents. The widest use of plain language is for documents that are intended for the general public. However, it is also applicable, for example, to technical writing, legislative drafting or using controlled languages. You don'…

> And you can read it for free here: https://www.iso.org/obp/ui#iso:std:iso:24495:-1:ed-1:v1:en

Actually, only the first few pages are available there (introductions, Sections 1-3.8). The meat of the document, Section 5 ("Guidelines") is completely absent.

See:

> Only informative sections of standards are publicly available. To view the full content, you will need to purchase the standard by clicking on the "Buy" button.

Re: Why does Opus 5 feel worse to work with?

#726

Post by DustinBrett »

dagmx wrote: post #649
Paradigma11 wrote: post #587

A gem Opus 5 gifted to me today: "A devastating pair of findings, and the first is beautiful in a way worth naming: the anti-vacuity floor is what blinds the gate to a vacuous case."

Oh yes, Claude seems to be loving the word vacuous recently. My test bed side project is full of vacuous this and that now. I even try and get it to define what it classifies as vacuous and it can’t do so without getting stuck in some kind of trap. It’s like a word with some kind of huge gravity for it.

Had same experience. I had to look up the usage as it just seemed so out of place.

Re: Why does Opus 5 feel worse to work with?

#727

Post by DustinBrett »

shrx wrote: post #672
Paradigma11 wrote: post #587

A gem Opus 5 gifted to me today: "A devastating pair of findings, and the first is beautiful in a way worth naming: the anti-vacuity floor is what blinds the gate to a vacuous case."

Two caveats, one of which matters:

You'll never guess which one...

Re: Why does Opus 5 feel worse to work with?

#728

Post by cma »

herbturbo wrote: post #479
mlsu wrote: post #428

Earlier quoted context omitted.

Everything that claude writes fits into the same aesthetic structure. The aesthetic is that of an expert slowly revealing an insight to the user. The actual content doesn't matter. - "Introduction that rephrases your prompt." - "3 paragraphs, with one section of bullet points" - "The Twist" - "The Bottom Line" It's really obvious once you see it. Every single prompt, from a quantum physics question to a mundane obser…

Perhaps this is related to their new "invisible watermark" concept which would probably require rather contrived language patterns to make possible.

If it's using Aaronson's approach it shouldn't have any noticeable affect on generations. When it picks between options weighted by probability after the generation of logits, it still follows the probability mass, it just uses a known pseudorandom seed so that when you go back and look at the exact choices you can fingerprint it.

Re: Why does Opus 5 feel worse to work with?

#729

Post by skzo »

sixdimensional wrote: post #305
repeekad wrote: post #267

Earlier quoted context omitted.

> owe it to society In America? lol if only, only a law would get them to act for that reason, maybe not even that these days..

Or, maybe competition.. but your point is taken. I like to hope that those in positions of power do have a sense of morality though too.. but their worldview is quite different than an ordinary citizen.

ROTFL

Re: Why does Opus 5 feel worse to work with?

#730

Post by geraneum »

zmmmmm wrote: post #674

I'm with the author and others in this comment thread, speculating that effectively the balance has tipped to where humans are no longer the target audience of post training - other agents are. Whether it's through the reasoning / CoT, or whether it's in handing off to subagents etc, the focus has moved to agents communicating in "agent-speak" to themselves or other agents. And human niceties are just kind of, noise…

> outcomes are evaluated purely empirically

How does that look like?

Post reply on HN