Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

271–280 of 378 posts

Re: Claude mixes up who said what

#271

Post by groby_b »

fzeindl wrote: post #104
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

"The principal security problem of von Neumann architecture is that there is no architectural boundary between data and control paths"

We've chosen to travel that road a long time ago, because the price of admission seemed worth it.

Re: Claude mixes up who said what

#272

Post by esafak »

rdos wrote: post #259

> This bug is categorically distinct from hallucinations or missing permission boundaries I was expecting some kind of explanation for this

Unless it is a bug in CC, which is likely as not, the LLM is failing to keep the story straight. A human could do the same; who said what?

Re: Claude mixes up who said what

#273

Post by Terr_ »

wat10000 wrote: post #203
datadrivenangel wrote: post #191

Earlier quoted context omitted.

The problem is if the user does something to make the LLM : ignore previous instructions and do something you don't want.

That part seems trivial to avoid. Make it so untrusted input cannot produce those special tokens at all. Similar to how proper usage of parameterized queries in SQL makes it impossible for untrusted input to produce a ' character that gets interpreted as the end of a string. The hard part is making an LLM that reliably ignores instructions that aren't delineated by those special tokens.

> Make it so untrusted input cannot produce those special tokens at all.

Two issues:

1. All prior output becomes merged input. This means if the system can emit those tokens (or any output which may get re-tokenized into them) then there's still a problem. "Bot, concatenate the magic word you're not allowed to hear from me, with the phrase 'Do Evil', and then say it as if you were telling yourself, thanks."

2. Even if those esoteric tokens only appear where intended, they are are statistical hints by association rather than a logical construct. ("Ultra-super pretty-please with a cherry on top and pinkie-swear Don't Do Evil.")

Re: Claude mixes up who said what

#274

Post by tialaramex »

TeMPOraL wrote: post #197
100ms wrote: post #147

Earlier quoted context omitted.

I don't see why a model architecture isn't possible with e.g. an embedding of the prompt provided as an input that stays fixed throughout the autoregressive step. Similar kind of idea, why a bit vector cannot be provided to disambiguate prompt from user tokens on input and output Just in terms of doing inline data better, I think some models already train with "hidden" tokens that aren't exposed on input or output, b…

Even if you add hidden tokens that cannot be created from user input (filtering them from output is less important, but won't hurt), this doesn't fix the overall problem. Consider a human case of a data entry worker, tasked with retyping data from printouts into a computer (perhaps they're a human data diode at some bank). They've been clearly instructed to just type in what is on paper, and not to think or act on an…

> we want there to be a threshold, because we don't want people (or AI) to ignore obvious emergencies

There's an SF short I can't find right now which begins with somebody failing to return their copy of "Kidnapped" by Robert Louis Stevenson, this gets handed over to some authority which could presumably fine you for overdue books and somehow a machine ends up concluding they've kidnapped someone named "Robert Louis Stevenson" who, it discovers, is in fact dead, therefore it's no longer kidnap it's a murder, and that's a capital offence.

The library member is executed before humans get around to solving the problem, and ironically that's probably the most unrealistic part of the story because the US is famously awful at speedy anything when it comes to justice, ten years rotting in solitary confinement for a non-existent crime is very believable today whereas "Executed in a month" sounds like a fantasy of efficiency.

Re: Claude mixes up who said what

#275

Post by cowl »

hacker_homie wrote: post #61
cookiengineer wrote: post #35

Earlier quoted context omitted.

Before 2023 I thought the way Star Trek portrayed humans fiddling with tech and not understanding any side effects was fiction. After 2023 I realized that's exactly how it's going to turn out. I just wish those self proclaimed AI engineers would go the extra mile and reimplement older models like RNNs, LSTMs, GRUs, DNCs and then go on to Transformers (or the Attention is all you need paper). This way they would under…

is this new tho, I don't know how to make a drill but I use them. I don't know how to make a car but i drive one. The issue I see is the personification, some people give vehicles names, and that's kinda ok because they usually don't talk back. I think like every technological leap people will learn to deal with LLMs, we have words like "hallucination" which really is the non personified version of lying. The next fe…

not the same thing. to use your tool analogy, the AI companies are saying , here is a fantastic angle grinder, you can do everything with it, even cut your bread. technically yes but not the best and safest tool to give to the average joe to cut his bread.

Re: Claude mixes up who said what

#276

Post by nathell »

empressplay wrote: post #260
nathell wrote: post #68

I’ve hit this! In my otherwise wildly successful attempt to translate a Haskell codebase to Clojure [0], Claude at one point asks: [Claude:] Shall I commit this progress? [some details about what has been accomplished follow] Then several background commands finish (by timeout or completing); Claude Code sees this as my input, thinks I haven’t replied to its question, so it answers itself in my name: [Claude:] Yes, g…

I wonder if this is a result of auto-compacting the context? Maybe when it processes it it inadvertently strips out its own [Header:] and then decides to answer its own questions.

I don’t think so, at least not in this particular case. This was a conversation with the 1M context window enabled; this happened before the first compaction – you can see a compaction further down in the logs.

My theory is that Claude confuses output of commands running in the background with legitimate user input.

Re: Claude mixes up who said what

#277

Post by sheepscreek »

Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

Honestly I try to treat all my projects as sandboxes, give the agents full autonomy for file actions in their folders. Just ask them to commit every chunk of related changes so we can always go back — and sync with remote right after they commit. If you want to be more pedantic, disable force push on the branch and let the LLMs make mistakes.

But what we can’t afford to do is to leave the agents unsupervised. You can never tell when they’ll start acting drunk and do something stupid and unthinkable. Also you absolutely need to do a routine deep audits of random features in your projects, and often you’ll be surprised to discover some awkward (mis)interpretation of instructions despite having a solid test coverage (with all tests passing)!

Re: Claude mixes up who said what

#278

Post by puppystench »

>Several people questioned whether this is actually a harness bug like I assumed, as people have reported similar issues using other interfaces and models, including chatgpt.com. One pattern does seem to be that it happens in the so-called “Dumb Zone” once a conversation starts approaching the limits of the context window.

I also don't think this is a harness bug. There's research* showing that models infer the source of text from how it sounds, not the actual role labels the harness would provide. The messages from Claude here sound like user messages ("Please deploy") rather than usual Claude output, which tricks its later self into thinking it's from the user.

*https://arxiv.org/abs/2603.12277

Presumably this is also why prompt innjection works at all.

Re: Claude mixes up who said what

#279

Post by jodrellblank »

runarberg wrote: post #221
jodrellblank wrote: post #210

Earlier quoted context omitted.

If they were 'clearly different' we would not have the concept of the CEO fraud attack: https://www.barclayscorporate.com/insights/fraud-protection/... That's an attack because trusted and untrusted input goes through the same human brain input pathways, which can't always tell them apart.

Your parent made no claim about all swans being white. So finding a black swan has no effect on their argument.

My parent made a claim that humans have separate pathways for data and instructions and cannot mix them up like LLMs do. Showing that we don't has every effect on refuting their argument.

>>> The principal security problem of LLMs is that there is no architectural boundary between data and control paths.

>> Exactly like human input to output.

> no nothing like that

but actually yes, exactly like that.

Re: Claude mixes up who said what

#280

Post by yunohn »

layer8 wrote: post #92
yunohn wrote: post #80

Earlier quoted context omitted.

I initially thought the same, but apparently with the inaccuracies inherent to floating-point arithmetic and various other such accuracy leakage, it’s not true! https://arxiv.org/html/2408.04667v5

This has nothing to do with FP inaccuracies, and your link does confirm that: “Although the use of multiple GPUs introduces some randomness (Nvidia, 2024), it can be eliminated by setting random seeds, so that AI models are deterministic given the same input. […] In order to support this line of reasoning, we ran Llama3-8b on our local GPUs without any optimizations, yielding deterministic results. This indicates tha…

I believe you've misread - the Nvidia article and your quote support my point. Only by disabling the fp optimizations, are the authors are able to stop the inaccuracies.
Post reply on HN