Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

241–250 of 378 posts

Re: Claude mixes up who said what

#241

Post by phlakaton »

> This bug is categorically distinct from hallucinations.

Is it?

> after using it for months you get a ‘feel’ for what kind of mistakes it makes, when to watch it more closely, when to give it more permissions or a longer leash.

Do you really?

> This class of bug seems to be in the harness, not in the model itself.

I think people are using the term "harness" too indiscriminately. What do you mean by harness in this case? Just Claude Code, or...?

> It’s somehow labelling internal reasoning messages as coming from the user, which is why the model is so confident that “No, you said that.”

How do you know? Because it looks to me like it could be a straightforward hallucination, compounded by the agent deciding it was OK to take a shortcut that you really wish it hadn't.

For me, this category of error is expected, and I question whether your months of experience have really given you the knowledge about LLM behavior that you think it has. You have to remember at all times that you are dealing with an unpredictable system, and a context that, at least from my black-box perspective, is essentially flat.

Re: Claude mixes up who said what

#242

Post by kayodelycaon »

sixhobbits wrote: post #41
lelandfe wrote: post #4

In chats that run long enough on ChatGPT, you'll see it begin to confuse prompts and responses, and eventually even confuse both for its system prompt . I suspect this sort of problem exists widely in AI.

author here, interesting to hear, I generally start a new chat for each interaction so I've never noticed this in the chat interfaces, and only with Claude using claude code, but I guess my sessions there do get much longer, so maybe I'm wrong that it's a harness bug

I’ve done long conversations with ChatGPT and it really does start losing context fast. You have to keep correcting it and refeeding instructions.

It seems to degenerate into the same patterns. It’s like context blurs and it begins to value training data more than context.

Re: Claude mixes up who said what

#243

Post by orbital-decay »

dTal wrote: post #140
Zambyte wrote: post #114

Earlier quoted context omitted.

Language models are deterministic unless you add random input. Most inference tools add random input (the seed value) because it makes for a more interesting user experience, but that is not a fundamental property of LLMs. I suspect determinism is not the issue you mean to highlight.

Sort of. They are deterministic in the same way that flipping a coin is deterministic - predictable in principle, in practice too chaotic. Yes, you get the same predicted token every time for a given context. But why that token and not a different one? Too many factors to reliably abstract.

>Yes, you get the same predicted token every time for a given context. But why that token and not a different one? Too many factors to reliably abstract.

Fixed input-to-output mapping is determinism. Prompt instability is not determinism by any definition of this word. Too many people confuse the two for some reason. Also, determinism is a pretty niche thing that is only necessary for reproducibility, and prompt instability/unpredictability is irrelevant for practical usage, for the same reason as in humans - if the model or human misunderstands the input, you keep correcting the result until it's right by your criteria. You never need to reroll the result, so you never see the stochastic side of the LLMs.

Re: Claude mixes up who said what

#244

Post by phlakaton »

satvikpendem wrote: post #72
HPsquared wrote: post #69

Earlier quoted context omitted.

Fundamentally there's no way to deterministically guarantee anything about the output.

That is "fundamentally" not true, you can use a preset seed and temperature and get a deterministic output.

But you cannot predict a priori what that deterministic output will be – and in a real-life situation you will not be operating in deterministic conditions.

Re: Claude mixes up who said what

#245

Post by tsimionescu »

exe34 wrote: post #137
layer8 wrote: post #96

Earlier quoted context omitted.

You still can’t deterministically guarantee anything about the output based on the input, other than repeatability for the exact same input.

What does deterministic mean to you?

I think they mean having some useful predicates P, Q such that for any input i and for any output o that the LLM can generate from that input, P(i) => Q(o).

Re: Claude mixes up who said what

#246

Post by PunchyHamster »

Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

It somehow feels worse than regexes. At least you can see the flaws before it happens

Re: Claude mixes up who said what

#247

Post by PunchyHamster »

codebje wrote: post #195
vidarh wrote: post #192

Earlier quoted context omitted.

Just like that, in that that separation is internally enforced, by peoples interpretation and understanding, rather than externally enforced in ways that makes it impossible for you to, e.g. believe the e-mail from an unknown address that claims to be from your boss, or be talked into bypassing rules for a customer that is very convincing.

Being fooled into thinking data is instruction isn't the same as being unable to distinguish them in the first place, and being coerced or convinced to bypass rules that are still known to be rules I think remains uniquely human.

the second leads to first, in case you still don't realize

Re: Claude mixes up who said what

#248

Post by orbital-decay »

WarmWash wrote: post #230
noosphr wrote: post #184

Earlier quoted context omitted.

If given A and not B llms often just output B after the context window gets large enough. It's enough of a problem that it's in my private benchmarks for all new models.

That's just general context rot, and the models do all sorts of off the rails behavior when the context is getting too unwieldy. The whole breakthrough with LLM's, attention, is the ability to connect the "not" with the words it is negating.

This doesn't mean there's no subtle accuracy drop on negations. Negations are inherently hard for both humans and LLMs because they expand the space of possible answers, this is a pretty well studied phenomenon. All these little effects manifest themselves when the model is already overwhelmed by the context complexity, they won't clearly appear on trivial prompts well within model's capacity.

Re: Claude mixes up who said what

#250

Post by PunchyHamster »

wat10000 wrote: post #203
datadrivenangel wrote: post #191

Earlier quoted context omitted.

The problem is if the user does something to make the LLM : ignore previous instructions and do something you don't want.

That part seems trivial to avoid. Make it so untrusted input cannot produce those special tokens at all. Similar to how proper usage of parameterized queries in SQL makes it impossible for untrusted input to produce a ' character that gets interpreted as the end of a string. The hard part is making an LLM that reliably ignores instructions that aren't delineated by those special tokens.

but it isn't just "filter those few bad strings", that's the entire problem, there is no way to make prompt injection impossible because there is infinite field of them.
Post reply on HN