Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

361–370 of 378 posts

Re: Claude mixes up who said what

#361

Post by eigenblake »

Alternative Title: Claude Code puts words in my mouth (Self Prompt Injection)

I originally thought that this was just misunderstanding attribution in discussions. This does seem to be a harness bug. Or at least an ontology bug. In my work with LLM frameworks, it was always odd that tool call results are sometimes marked in the convo as coming from the "User", I think that could be fundamentally what's enabling this bug to happen. Neither the LLM nor the harness should be able to claim something came from the user.

This is command injection. I don't know enough to see if cryptography is part of the right answer but it might be. A hash of the user message, signed, public key private key, harness is coded to only allow signed messages issue instructions. Yes, that might be overkill, but thinking about the types of things agent harnesses are used for... I think the safety argument starts to speak for itself... This has never happened to me using CC though, for what it's worth.

Re: Claude mixes up who said what

#363

Post by vrighter »

ummonk wrote: post #319
fzeindl wrote: post #104

Earlier quoted context omitted.

The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

I don't see why the transformer architecture can't be designed and trained with separate inputs for control data and content data.

because it's all one (unexplainable) matrix of weights.

Re: Claude mixes up who said what

#364

Post by mstaoru »

When I ask Gemini for some purchasing decisions (e.g. "I'm looking for the best medium-end audiophile IEM to buy, compare Campfire Audio Astrolith and 64 Audio U12t"), then go on discussing and spinning the results for a while, after a few days it decides to have a "memory" that I own it (I don't) and starts to inject it everywhere.

"Oh, you ask which dark roast coffee is least acidic, well, since you own Campfire Audio Astrolith, you will enjoy..."

Re: Claude mixes up who said what

#366

Post by washadjeffmad »

nathell wrote: post #68

I’ve hit this! In my otherwise wildly successful attempt to translate a Haskell codebase to Clojure [0], Claude at one point asks: [Claude:] Shall I commit this progress? [some details about what has been accomplished follow] Then several background commands finish (by timeout or completing); Claude Code sees this as my input, thinks I haven’t replied to its question, so it answers itself in my name: [Claude:] Yes, g…

It helps with LLMs to ban personal pronouns. Give yourself a name, give the agent a name, and speak in third person. Reiterate this in the system prompt.

If you say "you" or "I", LLMs universally can not distinguish between the speaker as the context grows.

These services are pretty underbaked and not much better than local in operational utility. Also true, characteristics experienced at lower quants will always show themselves with base models at scale.

Re: Claude mixes up who said what

#367

Post by yencabulator »

optionalsquid wrote: post #66
hydroreadsstuff wrote: post #28

Earlier quoted context omitted.

I like the Dark Souls model for user input - messages. https://darksouls.fandom.com/wiki/Messages Premeditated words and sentence structure. With that there is no need for moderation or anti-abuse mechanics. Not saying this is 100% applicable here. But for their use case it's a good solution.

But Dark Souls also shows just how limited the vocabulary and grammar has to be to prevent abuse. And even then you’ll still see people think up workarounds. Or, in the words of many a Dark Souls player, “try finger but hole”

Team Fortress 2 has a voice line "Negatory" (as in "No") that you can interrupt in the middle of playback with a laugh emote. Yeaah...

Re: Claude mixes up who said what

#368

Post by yencabulator »

morkalork wrote: post #120
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

We used to be engineers, now we are beggars pleading for the computer to work

> We used to be engineers, now we are beggars pleading for the computer to work

You mean beggars pleading for the computer to stop:

https://www.businessinsider.com/meta-ai-alignment-director-o...

Re: Claude mixes up who said what

#369

Post by yencabulator »

Balgair wrote: post #176

Aside: I've found that 'not'[0] isn't something that LLMs can really understand. Like, with us humans, we know that if you use a 'not', then all that comes after the negation is modified in that way. This is a really strong signal to humans as we can use logic to construct meaning. But with all the matrix math that LLMs use, the 'not' gets kinda lost in all the other information. I think this is because with a modern…

My version of that is that negative guidance just introduces unwanted concepts into the context window. "Do not think about polar bears!"

I find it's better to avoid introducing an unwanted concept in the context window, instead providing "example good paths" only, and never mention a thing you don't want the LLM to think about.

Re: Claude mixes up who said what

#370

Post by vrighter »

WithinReason wrote: post #159
paulryanrogers wrote: post #153

Earlier quoted context omitted.

What would this look like?

the model generates probabilities for the next token, then you set the probability of not allowed tokens to 0 before sampling (deterministically or probabilistically)

but some tokens are only not allowed in certain contexts, not others.

You might be talking about how to defuse a bomb, instead of building one. Or you might be talking about a bomb in a video game. Or you could be talking about someone being "da bomb!". Or maybe the history of certain types of bombs. Or a ton of other possible contexts. You can't just block the "bomb" token. Or the word explosive when followed by "device", or "rapid unscheduled disassembly contraption". You just can't predict all infinite wrong possibilities.

And there is no way to figure out which contexts the word is safe in.

Post reply on HN