Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

301–310 of 378 posts

Re: Claude mixes up who said what

#301

Post by SkyBelow »

100ms wrote: post #147
TeMPOraL wrote: post #130

Earlier quoted context omitted.

I've been saying this for a while, the issue is that what you're asking for is not possible, period. Prompt injection isn't like SQL injection, it's like social engineering - you can't eliminate it without also destroying the very capabilities you're using a general-purpose system for in the first place, whether that's an LLM or a human. It's not a bug, it's the feature.

I don't see why a model architecture isn't possible with e.g. an embedding of the prompt provided as an input that stays fixed throughout the autoregressive step. Similar kind of idea, why a bit vector cannot be provided to disambiguate prompt from user tokens on input and output Just in terms of doing inline data better, I think some models already train with "hidden" tokens that aren't exposed on input or output, b…

You can try to set up a NN where some of the neurons are either only activated off of 'safe' input (directly or indirectly from other 'safe' neurons), but as some point the information from them will have to flow over into the main output neurons which are also activating off unsafe user input. Where the information combines is there the user's input can corrupt whatever info comes from the safe input. There are plenty of attempts to make it less likely, but at the point of combining, there is a mixing of sources that can't fully be separated. It isn't that these don't help, but that they can't guarantee safety.

Then again, ever since the first von Neumann machine mixed data and instructions, we were never able to again guarantee safely splitting them. Is there any computer connected to the internet that is truly unhackable?

Re: Claude mixes up who said what

#302

Post by andai »

Yeah, GPT also constantly misattributes things.

OpenAI have some kinda 5 tier content hierarchy for OpenAI (system prompt, user prompt, untrusted web content etc). But if it doesn't even know who said what, I have to question how well that works.

Maybe it's trained on the security aspects, but not the attribution because there's no reward function for misattribution? (When it doesn't impact security or benchmark scores.)

Re: Claude mixes up who said what

#303

Post by andai »

Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

I tried to get GPT to talk like a regular guy yesterday. It was impossible for it to maintain adherence. It kept defaulting back to markdown and bullet points, after the first message. (Funny cause it scores highest on the instruction following benchmarks.)

Might seem trivial but if it can't even do a basic style prompt... how are you supposed to trust it with anything serious?

Re: Claude mixes up who said what

#304

Post by gaigalas »

> the so-called “Dumb Zone” once a conversation starts approaching the limits of the context window.

My zipper would totally break at some point very close to the edge of the mechanism. However, there is a little tiny stopper that prevents a bad experience.

If there is indeed a problem with context window tolerances, it should have a stopper. And the models should be sold based on their actual tolerances, not the full window considering the useless part.

So, if a model with 1M context window starts to break down consistently at 400K or so, it should be sold as a 400K model instead, with a 400K price.

The fact that it isn't is just dishonest.

Re: Claude mixes up who said what

#307

Post by Latty »

notatoad wrote: post #267
fzeindl wrote: post #104

Earlier quoted context omitted.

The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

As the article says: this doesn’t necessarily appear to be a problem in the LLM, it’s a problem in Claude code. Claude code seems to leave it up to the LLM to determine what messages came from who, but it doesn’t have to do that. There is a deterministic architectural boundary between data and control in Claude code, even if there isn’t in Claude.

That's a guess by the article author and frankly I see no supporting evidence for it. Wrapping "" tags around it or whatever is what I'm describing: you can do as much signalling as you want, but at the end of the day the LLM can ignore it.

Re: Claude mixes up who said what

#308

Post by [deleted] »

salt4034 wrote: post #288
bcrosby95 wrote: post #270

Earlier quoted context omitted.

This is where the old line of "LLMs are just next token predictors" actually factors in. I don't know how you get a next token predictor that user input can't break out of. The answer is for the implementer to try to split what they can, and run pre/post validation. But I highly doubt it will ever be 100%, its fundamental to the technology.

It's hard in general, but for instruct/chat models in particular, which already assume a turn-based approach, could they not use a special token that switches control from LLM output to user input? The LLM architecture could be made so it's literally impossible for the model to even produce this token. In the example above, the LLM could then recognize this is not a legitimate user input, as it lacks the token. I'm p…

[deleted]

Re: Claude mixes up who said what

#309

Post by themafia »

orbital-decay wrote: post #211

Claude in particular has nothing to do with it. I see many people are discovering the well-known fundamental biases and phenomena in LLMs again and again. There are many of those. The best intuition is treating the context as "kind of but not quite" an associative memory, instead of a sequence or a text file with tokens. This is vaguely similar to what humans are good and bad at, and makes it obvious what is easy and…

So easy it should disqualify you if you fail this: Knowing your own name.

Re: Claude mixes up who said what

#310

Post by noosphr »

WarmWash wrote: post #230
noosphr wrote: post #184

Earlier quoted context omitted.

If given A and not B llms often just output B after the context window gets large enough. It's enough of a problem that it's in my private benchmarks for all new models.

That's just general context rot, and the models do all sorts of off the rails behavior when the context is getting too unwieldy. The whole breakthrough with LLM's, attention, is the ability to connect the "not" with the words it is negating.

Large enough is usually between 5 to 10% of the advertised context.
Post reply on HN