Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

221–230 of 378 posts

Re: Claude mixes up who said what

#221

Post by runarberg »

jodrellblank wrote: post #210
codebje wrote: post #170

Earlier quoted context omitted.

Well no, nothing like that, because customers and bosses are clearly different forms of interaction.

If they were 'clearly different' we would not have the concept of the CEO fraud attack: https://www.barclayscorporate.com/insights/fraud-protection/... That's an attack because trusted and untrusted input goes through the same human brain input pathways, which can't always tell them apart.

Your parent made no claim about all swans being white. So finding a black swan has no effect on their argument.

Re: Claude mixes up who said what

#222

Post by vannevar »

morkalork wrote: post #120
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

We used to be engineers, now we are beggars pleading for the computer to work

I don't know, "pleading for the computer to work" pretty much sums up my entire 40-year career in software. Only the level of abstraction has changed.

Re: Claude mixes up who said what

#223

Post by benhurmarcel »

insin wrote: post #20
lelandfe wrote: post #4

In chats that run long enough on ChatGPT, you'll see it begin to confuse prompts and responses, and eventually even confuse both for its system prompt . I suspect this sort of problem exists widely in AI.

Gemini seems to be an expert in mistaking its own terrible suggestions as written by you, if you keep going instead of pruning the context

In Gemini chat I find that you should avoid continuing a conversation if its answer was wrong or had a big shortcoming. It's better to edit the previous prompt so that it comes up with a better answer in the first place, instead of sending a new message.

Re: Claude mixes up who said what

#224

Post by easeout »

__alexs wrote: post #8

Why are tokens not coloured? Would there just be too many params if we double the token count so the model could always tell input tokens from output tokens?

Because they're the main prompt injection vector, I think you'd want to distinguish tool results from user messages. By the time you go that far, you need colors for those two, plus system messages, plus thinking/responses. I have to think it's been tried and it just cost too much capability but it may be the best opportunity to improve at some point.

Re: Claude mixes up who said what

#225

Post by qsera »

TeMPOraL wrote: post #197
100ms wrote: post #147

Earlier quoted context omitted.

I don't see why a model architecture isn't possible with e.g. an embedding of the prompt provided as an input that stays fixed throughout the autoregressive step. Similar kind of idea, why a bit vector cannot be provided to disambiguate prompt from user tokens on input and output Just in terms of doing inline data better, I think some models already train with "hidden" tokens that aren't exposed on input or output, b…

Even if you add hidden tokens that cannot be created from user input (filtering them from output is less important, but won't hurt), this doesn't fix the overall problem. Consider a human case of a data entry worker, tasked with retyping data from printouts into a computer (perhaps they're a human data diode at some bank). They've been clearly instructed to just type in what is on paper, and not to think or act on an…

>If you were there, what would you do?

Show it to my boss and let them decide.

Re: Claude mixes up who said what

#226

Post by jerf »

xg15 wrote: post #33
__alexs wrote: post #8

Why are tokens not coloured? Would there just be too many params if we double the token count so the model could always tell input tokens from output tokens?

That's something I'm wondering as well. Not sure how it is with frontier models, but what you can see on Huggingface, the "standard" method to distinguish tokens still seems to be special delimiter tokens or even just formatting. Are there technical reasons why you can't make the "source" of the token (system prompt, user prompt, model thinking output, model response output, tool call, tool result, etc) a part of the…

By the nature of the LLM architecture I think if you "colored" the input via tokens the model would about 85% "unlearn" the coloring anyhow. Which is to say, it's going to figure out that "test" in the two different colors is the same thing. It kind of has to, after all, you don't want to be talking about a "test" in your prompt and it be completely unable to connect that to the concept of "test" in its own replies. The coloring would end up as just another language in an already multi-language model. It might slightly help but I doubt it would be a solution to the problem. And possibly at an unacceptable loss of capability as it would burn some of its capacity on that "unlearning".

Re: Claude mixes up who said what

#227

Post by kg »

vidarh wrote: post #217
codebje wrote: post #195

Earlier quoted context omitted.

Being fooled into thinking data is instruction isn't the same as being unable to distinguish them in the first place, and being coerced or convinced to bypass rules that are still known to be rules I think remains uniquely human.

This makes no sense to me. Being fooled into thinking data is instruction is exactly evidence of an inability to reliably distinguish them. And being coerced or convinced to bypass rules is exactly what prompt injection is, and very much not uniquely human any more.

The email from your boss and the email from a sender masquerading as your boss are both coming through the same channel in the same format with the same presentation, which is why the attack works. Unless you were both faceblind and bad at recognizing voices, the same attack wouldn't work in-person, you'd know the attacker wasn't your boss. Many defense mechanisms used in corporate email environments are built around making sure the email from your boss looks meaningfully different in order to establish that data vs instruction separation. (There are social engineering attacks that would work in-person though, but I don't think it's right to equate those to LLM attacks.)

Prompt injection is just exploiting the lack of separation, it's not 'coercion' or 'convincing'. Though you could argue that things like jailbreaking are closer to coercion, I'm not convinced that a statistical token predictor can be coerced to do anything.

Re: Claude mixes up who said what

#228

Post by nunez »

varispeed wrote: post #119

One day Claude started saying odd things claiming they are from memory and I said them. It was telling me personal details of someone I don't know. Where the person lives, their children names, the job they do, experience, relationship issues etc. Eventually Claude said that it is sorry and that was a hallucination. Then he started doing that again. For instance when I asked it what router they'd recommend, they gone…

did you confirm whether the person was real or not? this is an absolutely massive breach of privacy if the person was real that's worth telling Anthropic about.

Re: Claude mixes up who said what

#229

Post by peyton »

layer8 wrote: post #168
exe34 wrote: post #137

Earlier quoted context omitted.

What does deterministic mean to you?

In this context, it means being able to deterministically predict properties of the output based on properties of the input. That is, you don’t treat each distinct input as a unicorn, but instead consider properties of the input, and you want to know useful properties of the output. With LLMs, you can only do that statistically at best, but not deterministically, in the sense of being able to know that whenever the i…

I mean can’t you have a grammar on both ends and just set out-of-language tokens to zero. I thought one of the APIs had a way to staple a JSON schema to the output, for ex.

We’re making pretty strong statements here. It’s not like it’s impossible to make sure DROP TABLE doesn’t get output.

Re: Claude mixes up who said what

#230

Post by WarmWash »

noosphr wrote: post #184
whycombinetor wrote: post #181

Earlier quoted context omitted.

Do you have evals for this claim? I don't really experience this

If given A and not B llms often just output B after the context window gets large enough. It's enough of a problem that it's in my private benchmarks for all new models.

That's just general context rot, and the models do all sorts of off the rails behavior when the context is getting too unwieldy.

The whole breakthrough with LLM's, attention, is the ability to connect the "not" with the words it is negating.

Post reply on HN