Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

101–110 of 378 posts

Re: Claude mixes up who said what

#101

Post by htrp »

hacker_homie wrote: post #64
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

whatever happened to the system prompt buffer? why did it not work out?

Re: Claude mixes up who said what

#102

Post by HeavyStorm »

Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The real issue is expecting an LLM to be deterministic when it's not.

Re: Claude mixes up who said what

#104

Post by fzeindl »

Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The principal security problem of LLMs is that there is no architectural boundary between data and control paths.

But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

Re: Claude mixes up who said what

#105

Post by hansmayer »

Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

"Make this application without bugs" :)

Re: Claude mixes up who said what

#106

Post by raincole »

exitb wrote: post #46
dtagames wrote: post #38

There is no separation of "who" and "what" in a context of tokens. Me and you are just short words that can get lost in the thread. In other words, in a given body of text, a piece that says "you" where another piece says "me" isn't different enough to trigger anything. Those words don't have the special weight they have with people, or any meaning at all, really.

Aren’t there some markers in the context that delimit sections? In such case the harness should prevent the model from creating a user block.

You misunderstood. The model doesn't create a user block here. The UI correctly shows what was user message and what was model response.

Re: Claude mixes up who said what

#107

Post by afc »

jasongi wrote: post #87
alkonaut wrote: post #70

Earlier quoted context omitted.

When you use LLMs with APIs I at least see the history as a json list of entries, each being tagged as coming from the user, the LLM or being a system prompt. So presumably (if we assume there isn't a bug where the sources are ignored in the cli app) then the problem is that encoding this state for the LLM isn' reliable. I.e. it get's what is effectively LLM said: thing A User said: thing B And it still manages to bl…

Someone correct me if I'm wrong, but an LLM does not interpret structured content like JSON. Everything is fed into the machine as tokens, even JSON. So your structure that says "human says foo" and "computer says bar" is not deterministically interpreted by the LLM as logical statements but as a sequence of tokens. And when the context contains a LOT of those sequences, especially further "back" in the window then t…

That's exactly my understanding as well. This is, essentially, the LLM hallucinating user messages nested inside its outputs. FWIWI I've seen Gemini do this frequently (especially on long agent loops).

Re: Claude mixes up who said what

#108

Post by __alexs »

nairboon wrote: post #97
__alexs wrote: post #85

Earlier quoted context omitted.

I think OpenAI and Anthropic probably have a lot of that lying around by now.

They have a lot of data in the form: user input, LLM output. Then the model learns what the previous LLM models produced, with all their flaws. The core LLM premise is that it learns from all available human text.

This hasn't been the full story for years now. All SOTA models are strongly post-trained with reinforcement learning to improve performance on specific problems and interaction patterns.

The vast majority of this training data is generated synthetically.

Re: Claude mixes up who said what

#109

Post by layer8 »

__alexs wrote: post #8

Why are tokens not coloured? Would there just be too many params if we double the token count so the model could always tell input tokens from output tokens?

This has the potential to improve things a lot, though there would still be a failure mode when the user quotes the model or the model (e.g. in thinking) quotes the user.

Re: Claude mixes up who said what

#110

Post by sanitycheck »

perching_aix wrote: post #47
sanitycheck wrote: post #32

Earlier quoted context omitted.

It's both, really. The companies selling us the service aren't saying "you should treat this LLM as a potentially hostile user on your machine and set up a new restricted account for it accordingly", they're just saying "download our app! connect it to all your stuff!" and we can't really blame ordinary users for doing that and getting into trouble.

There's a growing ecosystem of guardrailing methods, and these companies are contributing. Antrophic specifically puts in a lot of effort to better steer and characterize their models AFAIK. I primarily use Claude via VS Code, and it defaults to asking first before taking any action. It's simply not the wild west out here that you make it out to be, nor does it need to be. These are statistical systems, so issues can…

I'm a naturally paranoid, very detail-oriented, man who has been a professional software developer for >25 years. Do you know anyone who read the full terms and conditions for their last car rental agreement prior to signing anything? I did that.

I do not expect other people to be as careful with this stuff as I am, and my perception of risk comes not only from the "hang on, wtf?" feeling when reading official docs but also from seeing what supposedly technical users are talking about actually doing on Reddit, here, etc.

Of course I use Claude Code, I'm not a Luddite (though they had a point), but I don't trust it and I don't think other people should either.

Post reply on HN