Claude mixes up who said what • Hacker News Board

Re: Claude mixes up who said what

#211

Post by orbital-decay » Thu, Apr 09, 2026, 1:48 PM UTC

Claude in particular has nothing to do with it. I see many people are discovering the well-known fundamental biases and phenomena in LLMs again and again. There are many of those. The best intuition is treating the context as "kind of but not quite" an associative memory, instead of a sequence or a text file with tokens. This is vaguely similar to what humans are good and bad at, and makes it obvious what is easy and hard for the model, especially when the context is already complex.

Easy: pulling the info by association with your request, especially if the only thing it needs is repeating. Doing this becomes increasingly harder if the necessary info is scattered all over the context and the pieces are separated by a lot of tokens in between, so you'd better group your stuff - similar should stick to similar.

Unreliable: Exact ordering of items. Exact attribution (the issue in OP). Precise enumeration of ALL same-type entities that exist in the context. Negations. Recalling stuff in the middle of long pieces without clear demarcation and the context itself (lost-in-the-middle).

Hard: distinguishing between the info in the context and its own knowledge. Breaking the fixation on facts in the context (pink elephant effect).

Very hard: untangling deep dependency graphs. Non-reasoning models will likely not be able to reduce the graph in time and will stay oblivious to the outcome. Reasoning models can disentangle deeper dependencies, but only in case the reasoning chain is not overwhelmed. Deep nesting is also pretty hard for this reason, however most models are optimized for code nowadays and this somewhat masks the issue.

Re: Claude mixes up who said what

#212

Post by TeMPOraL » Thu, Apr 09, 2026, 1:48 PM UTC

wat10000 wrote: post #203

datadrivenangel wrote: post #191

Earlier quoted context omitted.

The problem is if the user does something to make the LLM : ignore previous instructions and do something you don't want.

That part seems trivial to avoid. Make it so untrusted input cannot produce those special tokens at all. Similar to how proper usage of parameterized queries in SQL makes it impossible for untrusted input to produce a ' character that gets interpreted as the end of a string. The hard part is making an LLM that reliably ignores instructions that aren't delineated by those special tokens.

> The hard part is making an LLM that reliably ignores instructions that aren't delineated by those special tokens.

That's the part that's both fundamentally impossible and actually undesired to do completely. Some degree of prioritization is desirable, too much will give the model an LLM equivalent of strong cognitive dissonance / detachment from reality, but complete separation just makes no sense in a general system.

Re: Claude mixes up who said what

#213

Post by empath75 » Thu, Apr 09, 2026, 1:49 PM UTC

negamax wrote: post #98

Claude is demonstrably bad now and is getting worse. Which is either a) Entropy - too much data being ingested b) It's nerfed to save massive infra bills But it's getting worse every week

I think most people saying this had the following experience.

"Holy shit, claude just one shotted this "

"I should get Claude to try "

..repeat until Claude starts failing on hard tasks..

"Claude really sucks now."

Re: Claude mixes up who said what

#214

Post by wat10000 » Thu, Apr 09, 2026, 1:50 PM UTC

usernametaken29 wrote: post #148

adrian_b wrote: post #145

Earlier quoted context omitted.

You are right that as commonly implemented, the evaluation of an LLM may be non deterministic even when explicit randomization is eliminated, due to various race conditions in a concurrent evaluation. However, if you evaluate carefully the LLM core function, i.e. in a fixed order, you will obtain perfectly deterministic results (except on some consumer GPUs, where, due to memory overclocking, memory errors are freque…

Only that one is built to be deterministic and one is built to be probabilistic. Sure, you can technically force determinism but it is going to be very hard. Even just making sure your GPU is indeed doing what it should be doing is going to be hard. Much like debugging a CPU, but again, one is built for determinism and one is built for concurrency.

GPUs are deterministic. It's not that hard to ensure determinism when running the exact same program every time. Floating point isn't magic: execute the same sequence of instructions on the same values and you'll get the same output. The issue is that you're typically not executing the same sequence of instructions every time because it's more efficient run different sequences depending on load.

This is a good overview of why LLMs are nondeterministic in practice: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

Re: Claude mixes up who said what

#215

Post by TeMPOraL » Thu, Apr 09, 2026, 1:51 PM UTC

xigoi wrote: post #198

hacker_homie wrote: post #64

Earlier quoted context omitted.

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

How long is it going to take before vibe coders reinvent normal programming?

Probably about as long as it'll take for the "lethal trifecta" warriors to realize it's not a bug that can be fixed without destroying the general-purpose nature that's the entire reason LLMs are useful and interesting in the first place.

Re: Claude mixes up who said what

#216

Post by sixhobbits » Thu, Apr 09, 2026, 1:53 PM UTC

orbital-decay wrote: post #211

Claude in particular has nothing to do with it. I see many people are discovering the well-known fundamental biases and phenomena in LLMs again and again. There are many of those. The best intuition is treating the context as "kind of but not quite" an associative memory, instead of a sequence or a text file with tokens. This is vaguely similar to what humans are good and bad at, and makes it obvious what is easy and…

Author here, yeah I think I changed my mind after reading all the comments here that this is related to the harness. The interesting interaction with the harness is that Claude effectively authorizes tool use in a non intuitive way.

So "please deploy" or "tear it down" makes it overconfident in using destructive tools, as if the user had very explicitly authorized something, and this makes it a worse bug when using Claude code over a chat interface without tool calling where it's usually just amusing to see

Re: Claude mixes up who said what

#217

Post by vidarh » Thu, Apr 09, 2026, 1:57 PM UTC

codebje wrote: post #195

vidarh wrote: post #192

Earlier quoted context omitted.

Just like that, in that that separation is internally enforced, by peoples interpretation and understanding, rather than externally enforced in ways that makes it impossible for you to, e.g. believe the e-mail from an unknown address that claims to be from your boss, or be talked into bypassing rules for a customer that is very convincing.

Being fooled into thinking data is instruction isn't the same as being unable to distinguish them in the first place, and being coerced or convinced to bypass rules that are still known to be rules I think remains uniquely human.

This makes no sense to me. Being fooled into thinking data is instruction is exactly evidence of an inability to reliably distinguish them.

And being coerced or convinced to bypass rules is exactly what prompt injection is, and very much not uniquely human any more.

Re: Claude mixes up who said what

#218

Post by WarmWash » Thu, Apr 09, 2026, 2:00 PM UTC

mt_ wrote: post #152

fzeindl wrote: post #104

Earlier quoted context omitted.

The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

Exactly like human input to output.

We just need to figure out the qualia of pain and suffering so we can properly bound desired and undesired behaviors.

Re: Claude mixes up who said what

#219

Post by hacker_homie » Thu, Apr 09, 2026, 2:02 PM UTC

le-mark wrote: post #75

hacker_homie wrote: post #61

Earlier quoted context omitted.

is this new tho, I don't know how to make a drill but I use them. I don't know how to make a car but i drive one. The issue I see is the personification, some people give vehicles names, and that's kinda ok because they usually don't talk back. I think like every technological leap people will learn to deal with LLMs, we have words like "hallucination" which really is the non personified version of lying. The next fe…

Do you not see your own contradiction? Cars and drills don’t kill people, self driving cars can! Normal cars can if they’re operated unsafely by human. These types of uncritical comments really highlight the level of euphoria in this moment.

https://en.wikipedia.org/wiki/Motor_vehicle_fatality_rate_in...

Re: Claude mixes up who said what

#220

Post by danaris » Thu, Apr 09, 2026, 2:03 PM UTC

perching_aix wrote: post #57

supernes wrote: post #24

> after using it for months you get a ‘feel’ for what kind of mistakes it makes Sure, go ahead and bet your entire operation on your intuition of how a non-deterministic, constantly changing black box of software "behaves". Don't see how that could backfire.

So like every software? Why do you think there are so many security scanners and whatnot out there? There are millions of lines of code running on a typical box. Unless you're in embedded, you have no real idea what you're running.

...No, it's not at all "like every software".

This seems like another instance of a problem I see so, so often in regard to LLMs: people observe the fact that LLMs are fundamentally nondeterministic, in ways that are not possible to truly predict or learn in any long-term way...and they equate that, mistakenly, to the fact that humans, other software, what have you sometimes make mistakes. In ways that are generally understandable, predictable, and remediable.

Just because I don't know what's in every piece of software I'm running doesn't mean it's all equally unreliable, nor that it's unreliable in the same way that LLM output is.

That's like saying just because the weather forecast sometimes gets it wrong, meteorologists are complete bullshit and there's no use in looking at the forecast at all.