Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

151–160 of 378 posts

Re: Claude mixes up who said what

#151

Post by clickety_clack »

fzeindl wrote: post #104
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

It’s easier not to have that separation, just like it was easier not to separate them before LLMs. This is architectural stuff that just hasn’t been figured out yet.

Re: Claude mixes up who said what

#152

Post by mt_ »

fzeindl wrote: post #104
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

Exactly like human input to output.

Re: Claude mixes up who said what

#153

Post by paulryanrogers »

WithinReason wrote: post #113
HPsquared wrote: post #69

Earlier quoted context omitted.

Fundamentally there's no way to deterministically guarantee anything about the output.

Of course there is, restrict decoding to allowed tokens for example

What would this look like?

Re: Claude mixes up who said what

#154

Post by hacker_homie »

htrp wrote: post #101
hacker_homie wrote: post #64

Earlier quoted context omitted.

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

whatever happened to the system prompt buffer? why did it not work out?

because it's a separate context window, it makes the model bigger, that space is not accessible to the "user". And the "language understanding" basically had to be done twice because it's a separate input to the transformer so you can't just toss a pile of text in there and say "figure it out".

so we are currently in the era of one giant context window.

Re: Claude mixes up who said what

#157

Post by otabdeveloper4 »

hansmayer wrote: post #105
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

"Make this application without bugs" :)

You forgot to add "you are a senior software engineer with PhD level architectural insights" though.

Re: Claude mixes up who said what

#158

Post by Kye »

Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

Modern LLMs do a great job of following instructions, especially when it comes to conflict between instructions from the prompter and attempts to hijack it in retrieval. Claude's models will even call out prompt injection attempts.

Right up until it bumps into the context window and compacts. Then it's up to how well the interface manages carrying important context through compaction.

Re: Claude mixes up who said what

#159

Post by WithinReason »

paulryanrogers wrote: post #153
WithinReason wrote: post #113

Earlier quoted context omitted.

Of course there is, restrict decoding to allowed tokens for example

What would this look like?

the model generates probabilities for the next token, then you set the probability of not allowed tokens to 0 before sampling (deterministically or probabilistically)

Re: Claude mixes up who said what

#160

Post by qeternity »

100ms wrote: post #147
TeMPOraL wrote: post #130

Earlier quoted context omitted.

I've been saying this for a while, the issue is that what you're asking for is not possible, period. Prompt injection isn't like SQL injection, it's like social engineering - you can't eliminate it without also destroying the very capabilities you're using a general-purpose system for in the first place, whether that's an LLM or a human. It's not a bug, it's the feature.

I don't see why a model architecture isn't possible with e.g. an embedding of the prompt provided as an input that stays fixed throughout the autoregressive step. Similar kind of idea, why a bit vector cannot be provided to disambiguate prompt from user tokens on input and output Just in terms of doing inline data better, I think some models already train with "hidden" tokens that aren't exposed on input or output, b…

This does not solve the problem at all, it's just another bandaid that hopefully reduces the likelihood.
Post reply on HN