Elevated errors on Claude Opus 5 • Hacker News Board

Re: Elevated errors on Claude Opus 5

#21

Post by sjsdaiuasgdia » Mon, Jul 27, 2026, 1:52 PM UTC

wongarsu wrote: post #12

rich_sasha wrote: post #3

I've noticed something else - as Anthropic models get even more and more superhuman, they seem to serve me more and more casual nonsense. Not like adding glue to pizza. Here's an example from today (paraphrasing): "you need to run `git merge-base branch1 branch2`. Pay attention to the order of arguments, it is important: `git merge-base` is symmetric and returns the same value regardless of the order of inputs". So w…

I've noticed the same pattern even in GLM5.2. It has always been a thing, but it seems to be getting worse in recent models It does feel like the kind of thing beam search would fix. The LLM starts the sentence with a claim like "Pay attention to the order of arguments". Around that time it "notices" that the order doesn't matter, but it's already committed to the sentence and has to complete it in the best way still…

Yes, LLMs are prone to semantic consistency traps. I see a version of this in a system I have that transcribes a lot of noisy low quality audio (radio comms).

Sometimes it works out, in that an unrecognizable word or two is replaced with reasonable assumptions based on the semantics established by the words that came before the signal degraded.

But the bad audio might also result in words that don't align well to what came before, or represent alternate (mis)interpretations. Now this is part of the context and the next several tokens align to this new path regardless of what is said in the audio.

In pre-LLM transcription, you might get a nonsense word or two when the audio transiently degrades, but the specific meaning of the nonsense words doesn't influence the transcription of audio following the degradation.

Re: Elevated errors on Claude Opus 5

#22

Post by maweaver » Mon, Jul 27, 2026, 1:54 PM UTC

wongarsu wrote: post #12

rich_sasha wrote: post #3

I've noticed something else - as Anthropic models get even more and more superhuman, they seem to serve me more and more casual nonsense. Not like adding glue to pizza. Here's an example from today (paraphrasing): "you need to run `git merge-base branch1 branch2`. Pay attention to the order of arguments, it is important: `git merge-base` is symmetric and returns the same value regardless of the order of inputs". So w…

I've noticed the same pattern even in GLM5.2. It has always been a thing, but it seems to be getting worse in recent models It does feel like the kind of thing beam search would fix. The LLM starts the sentence with a claim like "Pay attention to the order of arguments". Around that time it "notices" that the order doesn't matter, but it's already committed to the sentence and has to complete it in the best way still…

Have noticed similar things using it for code review. "Issue #3: This variable is defined but never used. It is actually used later in the method though. So this is not a real issue to be concerned about", that sort of thing.

Re: Elevated errors on Claude Opus 5

#23

Post by willmadden » Mon, Jul 27, 2026, 2:18 PM UTC

wongarsu wrote: post #12

rich_sasha wrote: post #3

I've noticed something else - as Anthropic models get even more and more superhuman, they seem to serve me more and more casual nonsense. Not like adding glue to pizza. Here's an example from today (paraphrasing): "you need to run `git merge-base branch1 branch2`. Pay attention to the order of arguments, it is important: `git merge-base` is symmetric and returns the same value regardless of the order of inputs". So w…

I've noticed the same pattern even in GLM5.2. It has always been a thing, but it seems to be getting worse in recent models It does feel like the kind of thing beam search would fix. The LLM starts the sentence with a claim like "Pay attention to the order of arguments". Around that time it "notices" that the order doesn't matter, but it's already committed to the sentence and has to complete it in the best way still…

Or a strikethrough token!

Re: Elevated errors on Claude Opus 5

#24

Post by ACCount37 » Mon, Jul 27, 2026, 4:43 PM UTC

rvz wrote: post #17

ACCount37 wrote: post #9

Earlier quoted context omitted.

Can we not do the "AI is drinking all the water" bullshit at least on HN?

There are many sources that already document it being true so we both know it is not bullshit. [0][1][2] So you think that the water that comes out of these data centers is safe for humans once released and the mass consumption of them is not a concern? Sounds like a way to sweep this environmental issue under the rug. [0] https://theoec.org/news-and-information/behind-the-data-boom... [1] https://fieldreport.caes.ug…

Sounds like a way to distract from the real environmental issues with yet another "plastic straws" debacle. A fake issue that makes for some nice headlines in the media - but when you run the numbers, it obviously doesn't matter and never did.

Distract from the real issues like, you know. Clean energy projects in the US being throttled on federal level due to lobbying by the oil companies - lobbying that borders on regulatory capture. At the time when the oil prices are rising globally - and the demand for energy in the US is rising too.

Unfortunately, you can't stop an environmentalist from throwing his weight behind the cause that's being promoted the loudest. So influence groups can just push one meaningless "headline cause" after another, and keep them distracted from anything that matters in perpetuity.

Re: Elevated errors on Claude Opus 5

#25

Post by bellowsgulch » Mon, Jul 27, 2026, 5:28 PM UTC

wongarsu wrote: post #12

rich_sasha wrote: post #3

I've noticed something else - as Anthropic models get even more and more superhuman, they seem to serve me more and more casual nonsense. Not like adding glue to pizza. Here's an example from today (paraphrasing): "you need to run `git merge-base branch1 branch2`. Pay attention to the order of arguments, it is important: `git merge-base` is symmetric and returns the same value regardless of the order of inputs". So w…

I've noticed the same pattern even in GLM5.2. It has always been a thing, but it seems to be getting worse in recent models It does feel like the kind of thing beam search would fix. The LLM starts the sentence with a claim like "Pay attention to the order of arguments". Around that time it "notices" that the order doesn't matter, but it's already committed to the sentence and has to complete it in the best way still…

First we started by extending chain-of-thought to a formal thinking system. You could extend this again with a "thought scratch-pad" or "thinking draft" with n-number of passes without needing to train backspace tokens.

That being said, all of these are just variations of scaffolding on generating more tokens.