Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

291–300 of 378 posts

Re: Claude mixes up who said what

#291

Post by puppystench »

xg15 wrote: post #13

> This class of bug seems to be in the harness, not in the model itself. It’s somehow labelling internal reasoning messages as coming from the user, which is why the model is so confident that “No, you said that.” Are we sure about this? Accidentally mis-routing a message is one thing, but those messages also distinctly "sound" like user messages, and not something you'd read in a reasoning trace. I'd like to know if…

I believe you're right, it's an issue of the model misinterpreting things that sound like user message as actual user messages. It's a known phenomenon: https://arxiv.org/abs/2603.12277

Re: Claude mixes up who said what

#292

Post by layer8 »

peyton wrote: post #229
layer8 wrote: post #168

Earlier quoted context omitted.

In this context, it means being able to deterministically predict properties of the output based on properties of the input. That is, you don’t treat each distinct input as a unicorn, but instead consider properties of the input, and you want to know useful properties of the output. With LLMs, you can only do that statistically at best, but not deterministically, in the sense of being able to know that whenever the i…

I mean can’t you have a grammar on both ends and just set out-of-language tokens to zero. I thought one of the APIs had a way to staple a JSON schema to the output, for ex. We’re making pretty strong statements here. It’s not like it’s impossible to make sure DROP TABLE doesn’t get output.

You still can’t predict whether the in-language responses will be correct or not.

As an analogy: If, for a compiler, you verify that its output is valid machine code, that doesn’t tell you whether the output machine code is faithful to the input source code. For example, you might want to have the assurance that if the input specifies a terminating program, then the output machine code represents a terminating program as well. For a compiler, you can guarantee that such properties are true by construction.

More generally, you can write your programs such that you can prove from their code that they satisfy properties you are interested in for all inputs.

With LLMs, however, you have no practical way to reason about relations between the properties of inputs and outputs.

Re: Claude mixes up who said what

#293

Post by lupire »

salt4034 wrote: post #288
bcrosby95 wrote: post #270

Earlier quoted context omitted.

This is where the old line of "LLMs are just next token predictors" actually factors in. I don't know how you get a next token predictor that user input can't break out of. The answer is for the implementer to try to split what they can, and run pre/post validation. But I highly doubt it will ever be 100%, its fundamental to the technology.

It's hard in general, but for instruct/chat models in particular, which already assume a turn-based approach, could they not use a special token that switches control from LLM output to user input? The LLM architecture could be made so it's literally impossible for the model to even produce this token. In the example above, the LLM could then recognize this is not a legitimate user input, as it lacks the token. I'm p…

Yes, and as you'd expect, this is how LLMs work today, in general, for control codes. But different elems use different control codes for different purposes, such as separating system prompt from user prompt.

But even if you tag inputs however your this is good, you can't force an LLM to it treat input type A as input type B, all you can do is try to weight against it! LLMs have no rules, only weights. Pre and post filters cam try to help, but they can't directly control the LLM text generation, they can only analyze and most inputs/output using their own heuristics.

Re: Claude mixes up who said what

#295

Post by spixy »

orbital-decay wrote: post #243
dTal wrote: post #140

Earlier quoted context omitted.

Sort of. They are deterministic in the same way that flipping a coin is deterministic - predictable in principle, in practice too chaotic. Yes, you get the same predicted token every time for a given context. But why that token and not a different one? Too many factors to reliably abstract.

>Yes, you get the same predicted token every time for a given context. But why that token and not a different one? Too many factors to reliably abstract. Fixed input-to-output mapping is determinism. Prompt instability is not determinism by any definition of this word. Too many people confuse the two for some reason. Also, determinism is a pretty niche thing that is only necessary for reproducibility, and prompt inst…

But there is no fixed input-to-output mapping in current popuular LLMs.

Re: Claude mixes up who said what

#297

Post by pessimizer »

All of the models that I've used do this. They, extremely often, pretend to have corrected me right after I've corrected them. Verbosely. Feeding my own correction back to me as a correction of my mistake.

Even when they don't forget who corrected who, often their taking in the correction also just involves feeding the exact words of my correction back to me rather than continuing to solve the problem using the correction. Honestly, the context is poisoned by then and it's forgotten the problem anyway.

Of course it's forgotten the problem; how stupid would you have to be to think that I wanted an extensive recap of the correction I just gave it rather than my problem solved (even without the confusion)? Best case scenario:

Me: Hand me the book.

Machine: [reaches for the top shelf]

Me: [sees you reach for the top shelf] No, it's on the bottom shelf.

Machine: When you asked for the book, I reached for the top shelf, then you said that it was on the bottom shelf, and it's more than fair that you hold me to that standard, the book is on the bottom shelf.

(Or, half the time: "You asked me to get the book from the top shelf, but no, it's on the bottom shelf.")

Machine: [sits down]

Me: Booooooooooook. GET THE BOOK. GET THE BOOK.

These things are so dumb. I'm begging for somebody to show me the sequence that makes me feel the sort of threat that they seem to feel. They're mediocre at writing basic code (which is still mind-blowing and super-helpful), and they have all the manuals and docs in their virtual heads (and all the old versions cause them to constantly screw up and hallucinate.) But other than that...

Re: Claude mixes up who said what

#298

Post by warkdarrior »

lupire wrote: post #290
miki123211 wrote: post #287

Earlier quoted context omitted.

I think this is fundamental to any technology, including human brains. Humans have a problem distinguishing "John from Microsoft" from somebody just claiming to be John from Microsoft. The reason why scamming humans is (relatively) hard is that each human is different. Discovering the perfect tactic to scam one human doesn't necessarily scale across all humans. LLMs are the opposite; my Chat GPT is (almost) the same…

How are you defining "banner blindness"? The foundation of LLMs is Attention.

"Banner blindness [...] describes people’s tendency to ignore page elements that they perceive (correctly or incorrectly) to be ads." https://www.nngroup.com/articles/banner-blindness-old-and-ne...

So people can focus their attention to parts of content, specifically parts they find irrelevant or adversarial (like ads). LLMs on the other hand pay attention to everything or if they focus on something, it is hard to steer them away from irrelevant or adversarial parts.

Re: Claude mixes up who said what

#299

Post by [deleted] »

salt4034 wrote: post #288
bcrosby95 wrote: post #270

Earlier quoted context omitted.

This is where the old line of "LLMs are just next token predictors" actually factors in. I don't know how you get a next token predictor that user input can't break out of. The answer is for the implementer to try to split what they can, and run pre/post validation. But I highly doubt it will ever be 100%, its fundamental to the technology.

It's hard in general, but for instruct/chat models in particular, which already assume a turn-based approach, could they not use a special token that switches control from LLM output to user input? The LLM architecture could be made so it's literally impossible for the model to even produce this token. In the example above, the LLM could then recognize this is not a legitimate user input, as it lacks the token. I'm p…

[deleted]

Re: Claude mixes up who said what

#300

Post by orbital-decay »

spixy wrote: post #295
orbital-decay wrote: post #243

Earlier quoted context omitted.

>Yes, you get the same predicted token every time for a given context. But why that token and not a different one? Too many factors to reliably abstract. Fixed input-to-output mapping is determinism. Prompt instability is not determinism by any definition of this word. Too many people confuse the two for some reason. Also, determinism is a pretty niche thing that is only necessary for reproducibility, and prompt inst…

But there is no fixed input-to-output mapping in current popuular LLMs.

You mean "corporate inference infrastructure", not LLMs. The reason for different outputs at t=0 is mostly batching optimization. LLMs themselves are indifferent to that, you can run them in a deterministic manner any time if you don't care about optimal batching and lowest possible inference cost. And even then, e.g. Gemini Flash is deterministic in practice even with batching, although DeepMind doesn't strictly guarantee it.

This is all currently irrelevant, making it work well is a much bigger problem. As soon as there's paying demand for reproducibility, solutions will appear. This is a matter of business need, not a technical issue.

Post reply on HN