Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

111–120 of 378 posts

Re: Claude mixes up who said what

#111

Post by yanis_t »

xg15 wrote: post #13

> This class of bug seems to be in the harness, not in the model itself. It’s somehow labelling internal reasoning messages as coming from the user, which is why the model is so confident that “No, you said that.” Are we sure about this? Accidentally mis-routing a message is one thing, but those messages also distinctly "sound" like user messages, and not something you'd read in a reasoning trace. I'd like to know if…

Also could be a bit both, with harness constructing context in a way that model misinterprets it.

Re: Claude mixes up who said what

#112

Post by WithinReason »

HeavyStorm wrote: post #102
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The real issue is expecting an LLM to be deterministic when it's not.

Oh how I wish people understood the word "deterministic"

Re: Claude mixes up who said what

#113

Post by WithinReason »

HPsquared wrote: post #69
hacker_homie wrote: post #64

Earlier quoted context omitted.

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

Fundamentally there's no way to deterministically guarantee anything about the output.

Of course there is, restrict decoding to allowed tokens for example

Re: Claude mixes up who said what

#114

Post by Zambyte »

HeavyStorm wrote: post #102
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The real issue is expecting an LLM to be deterministic when it's not.

Language models are deterministic unless you add random input. Most inference tools add random input (the seed value) because it makes for a more interesting user experience, but that is not a fundamental property of LLMs. I suspect determinism is not the issue you mean to highlight.

Re: Claude mixes up who said what

#115

Post by baq »

HeavyStorm wrote: post #102
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The real issue is expecting an LLM to be deterministic when it's not.

LLMs are essentially pure functions.

Re: Claude mixes up who said what

#116

Post by scotty79 »

lelandfe wrote: post #4

In chats that run long enough on ChatGPT, you'll see it begin to confuse prompts and responses, and eventually even confuse both for its system prompt . I suspect this sort of problem exists widely in AI.

It makes sense. It's all probabilistic and it all gets fuzzy when garbage in context accumulates. User messages or system prompt got through the same network of math as model thinking and responses.

Re: Claude mixes up who said what

#117

Post by WithinReason »

idiotsecant wrote: post #89
satvikpendem wrote: post #82

Earlier quoted context omitted.

Well yeah of course changes in the input result in changes to the output, my only claim was that LLMs can be deterministic (ie to output exactly the same output each time for a given input) if set up correctly.

You don't think this is pedantry bordering on uselessness?

No, determinism and predictability are different concepts. You can have a deterministic random number generator for example.

Re: Claude mixes up who said what

#118

Post by coffeefirst »

jasongi wrote: post #87
alkonaut wrote: post #70

Earlier quoted context omitted.

When you use LLMs with APIs I at least see the history as a json list of entries, each being tagged as coming from the user, the LLM or being a system prompt. So presumably (if we assume there isn't a bug where the sources are ignored in the cli app) then the problem is that encoding this state for the LLM isn' reliable. I.e. it get's what is effectively LLM said: thing A User said: thing B And it still manages to bl…

Someone correct me if I'm wrong, but an LLM does not interpret structured content like JSON. Everything is fed into the machine as tokens, even JSON. So your structure that says "human says foo" and "computer says bar" is not deterministically interpreted by the LLM as logical statements but as a sequence of tokens. And when the context contains a LOT of those sequences, especially further "back" in the window then t…

I think that’s correct. There seems to be a lot of fundamental limitations that have been “fixed” through a boatload of reinforcement learning.

But that doesn’t make them go away, it just makes them less glaring.

Re: Claude mixes up who said what

#119

Post by varispeed »

One day Claude started saying odd things claiming they are from memory and I said them. It was telling me personal details of someone I don't know. Where the person lives, their children names, the job they do, experience, relationship issues etc. Eventually Claude said that it is sorry and that was a hallucination. Then he started doing that again. For instance when I asked it what router they'd recommend, they gone on saying: "Since you bought X and you find no use for it, consider turning it into a router". I said I never told you I bought X and I asked for more details and it again started coming up what this guy did. Strange. Then again it apologised saying that it might be unsettling, but rest assured that is not a leak of personal information, just hallucinations.

Re: Claude mixes up who said what

#120

Post by morkalork »

Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

We used to be engineers, now we are beggars pleading for the computer to work
Post reply on HN