Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

131–140 of 378 posts

Re: Claude mixes up who said what

#131

Post by usernametaken29 »

Zambyte wrote: post #114
HeavyStorm wrote: post #102

Earlier quoted context omitted.

The real issue is expecting an LLM to be deterministic when it's not.

Language models are deterministic unless you add random input. Most inference tools add random input (the seed value) because it makes for a more interesting user experience, but that is not a fundamental property of LLMs. I suspect determinism is not the issue you mean to highlight.

Actually at a hardware level floating point operations are not associative. So even with temperature of 0 you’re not mathematically guaranteed the same response. Hence, not deterministic.

Re: Claude mixes up who said what

#132

Post by cmrdporcupine »

spprashant wrote: post #121
hacker_homie wrote: post #64

Earlier quoted context omitted.

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

The problem is once you accept that it is needed, you can no longer push AI as general intelligence that has superior understanding of the language we speak. A structured LLM query is a programming language and then you have to accept you need software engineers for sufficiently complex structured queries. This goes against everything the technocrats have been saying.

Perhaps, though it's not infeasible the concept that you could have a small and fast general purpose language focused model in front whose job it is to convert English text into some sort of more deterministic propositional logic "structured LLM query" (and back).

Re: Claude mixes up who said what

#133

Post by adrian_b »

dwattttt wrote: post #94
simianparrot wrote: post #79

Earlier quoted context omitted.

A single byte change in the input changes the output. The sentence "Please do this for me" and "Please, do this for me" can lead to completely distinct output. Given this, you can't treat it as deterministic even with temp 0 and fixed seed and no memory.

Interestingly, this is the mathematical definition of "chaotic behaviour"; minuscule changes in the input result in arbitrarily large differences in the output. It can arise from perfectly deterministic rules... the Logistic Map with r=4, x(n+1) = 4*(1 - x(n)) is a classic.

Which is also the desired behavior of the mixing functions from which the cryptographic primitives are built (e.g. block cipher functions and one-way hash functions), i.e. the so-called avalanche property.

Re: Claude mixes up who said what

#134

Post by exe34 »

simianparrot wrote: post #79
satvikpendem wrote: post #72

Earlier quoted context omitted.

That is "fundamentally" not true, you can use a preset seed and temperature and get a deterministic output.

A single byte change in the input changes the output. The sentence "Please do this for me" and "Please, do this for me" can lead to completely distinct output. Given this, you can't treat it as deterministic even with temp 0 and fixed seed and no memory.

Let's eat grandma.

Re: Claude mixes up who said what

#135

Post by 8note »

bravetraveler wrote: post #122
ares623 wrote: post #76

Earlier quoted context omitted.

I wonder if tools like Terraform should remove the message "Run terraform apply plan.out next" that it prints after every `terraform plan` is run.

I don't think so, feels like the wrong side is getting attention. Degrading the experience for humans (in one tool) because the bots are prone to injection (from any tool) . Terraform is used outside of agents; somebody surely finds the reminder helpful. If terraform were to abide, I'd hope at the very least it would check if in a pipeline or under an agent. This should be obvious from file descriptors/env. What abou…

it makes you wonder how many times people have incorrectly followed those recommended commands

Re: Claude mixes up who said what

#136

Post by TeMPOraL »

bravetraveler wrote: post #122
ares623 wrote: post #76

Earlier quoted context omitted.

I wonder if tools like Terraform should remove the message "Run terraform apply plan.out next" that it prints after every `terraform plan` is run.

I don't think so, feels like the wrong side is getting attention. Degrading the experience for humans (in one tool) because the bots are prone to injection (from any tool) . Terraform is used outside of agents; somebody surely finds the reminder helpful. If terraform were to abide, I'd hope at the very least it would check if in a pipeline or under an agent. This should be obvious from file descriptors/env. What abou…

"Run terraform apply plan.out next" in this context is a prompt injection for an LLM to exactly the same degree it is for a human.

Even a first party suggestion can be wrong in context, and if a malicious actor managed to substitute that message with a suggestion of their own, humans would fall for the trick even more than LLMs do.

See also: phishing.

Re: Claude mixes up who said what

#137

Post by exe34 »

layer8 wrote: post #96
satvikpendem wrote: post #82

Earlier quoted context omitted.

Well yeah of course changes in the input result in changes to the output, my only claim was that LLMs can be deterministic (ie to output exactly the same output each time for a given input) if set up correctly.

You still can’t deterministically guarantee anything about the output based on the input, other than repeatability for the exact same input.

What does deterministic mean to you?

Re: Claude mixes up who said what

#138

Post by codingdave »

hacker_homie wrote: post #64
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

That seems like an acceptable constraint to me. If you need a structured query, LLMs are the wrong solution. If you can accept ambiguity, LLMs may the the right solution.

Re: Claude mixes up who said what

#139

Post by bravetraveler »

TeMPOraL wrote: post #136
bravetraveler wrote: post #122

Earlier quoted context omitted.

I don't think so, feels like the wrong side is getting attention. Degrading the experience for humans (in one tool) because the bots are prone to injection (from any tool) . Terraform is used outside of agents; somebody surely finds the reminder helpful. If terraform were to abide, I'd hope at the very least it would check if in a pipeline or under an agent. This should be obvious from file descriptors/env. What abou…

"Run terraform apply plan.out next" in this context is a prompt injection for an LLM to exactly the same degree it is for a human. Even a first party suggestion can be wrong in context, and if a malicious actor managed to substitute that message with a suggestion of their own, humans would fall for the trick even more than LLMs do. See also: phishing.

Right, I'm fine with humans making the call. We're not so injection-happy/easily confused, apparently.

Discretion, etc. We understand that was the tool making a suggestion, not our idea. Our agency isn't in question.

The removal proposal is similar to wanting a phishing-free environment instead of preparing for the inevitability. I could see removing this message based on your point of context/utility, but not to protect the agent. We get no such protection, just training and practice.

A supply chain attack is another matter entirely; I'm sure people would pause at a new suggestion that deviates from their plan/training. As shown, autobots are eager to roll out and easily drown in context. So much so that `User` and `stdout` get confused.

Re: Claude mixes up who said what

#140

Post by dTal »

Zambyte wrote: post #114
HeavyStorm wrote: post #102

Earlier quoted context omitted.

The real issue is expecting an LLM to be deterministic when it's not.

Language models are deterministic unless you add random input. Most inference tools add random input (the seed value) because it makes for a more interesting user experience, but that is not a fundamental property of LLMs. I suspect determinism is not the issue you mean to highlight.

Sort of. They are deterministic in the same way that flipping a coin is deterministic - predictable in principle, in practice too chaotic. Yes, you get the same predicted token every time for a given context. But why that token and not a different one? Too many factors to reliably abstract.
Post reply on HN