Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

141–150 of 378 posts

Re: Claude mixes up who said what

#141

Post by curt15 »

HeavyStorm wrote: post #102
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The real issue is expecting an LLM to be deterministic when it's not.

LLMs are deterministic in the sense that a fixed linear regression model is deterministic. Like linear regression, however, they do however encode a statistical model of whatever they're trying to describe -- natural language for LLMs.

Re: Claude mixes up who said what

#143

Post by adam_patarino »

hacker_homie wrote: post #64
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

It’s not a query / prompt thing though is it? No matter the input LLMs rely on some degree of random. That’s what makes them what they are. We are just trying to force them into deterministic execution which goes against their nature.

Re: Claude mixes up who said what

#144

Post by bravetraveler »

8note wrote: post #135
bravetraveler wrote: post #122

Earlier quoted context omitted.

I don't think so, feels like the wrong side is getting attention. Degrading the experience for humans (in one tool) because the bots are prone to injection (from any tool) . Terraform is used outside of agents; somebody surely finds the reminder helpful. If terraform were to abide, I'd hope at the very least it would check if in a pipeline or under an agent. This should be obvious from file descriptors/env. What abou…

it makes you wonder how many times people have incorrectly followed those recommended commands

If more than once (individually), I am concerned.

Re: Claude mixes up who said what

#145

Post by adrian_b »

usernametaken29 wrote: post #131
Zambyte wrote: post #114

Earlier quoted context omitted.

Language models are deterministic unless you add random input. Most inference tools add random input (the seed value) because it makes for a more interesting user experience, but that is not a fundamental property of LLMs. I suspect determinism is not the issue you mean to highlight.

Actually at a hardware level floating point operations are not associative. So even with temperature of 0 you’re not mathematically guaranteed the same response. Hence, not deterministic.

You are right that as commonly implemented, the evaluation of an LLM may be non deterministic even when explicit randomization is eliminated, due to various race conditions in a concurrent evaluation.

However, if you evaluate carefully the LLM core function, i.e. in a fixed order, you will obtain perfectly deterministic results (except on some consumer GPUs, where, due to memory overclocking, memory errors are frequent, which causes slightly erroneous results with non-deterministic errors).

So if you want deterministic LLM results, you must audit the programs that you are using and eliminate the causes of non-determinism, and you must use good hardware.

This may require some work, but it can be done, similarly to the work that must be done if you want to deterministically build a software package, instead of obtaining different executable files at each recompilation from the same sources.

Re: Claude mixes up who said what

#147

Post by 100ms »

TeMPOraL wrote: post #130
hacker_homie wrote: post #64

Earlier quoted context omitted.

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

I've been saying this for a while, the issue is that what you're asking for is not possible, period. Prompt injection isn't like SQL injection, it's like social engineering - you can't eliminate it without also destroying the very capabilities you're using a general-purpose system for in the first place, whether that's an LLM or a human. It's not a bug, it's the feature.

I don't see why a model architecture isn't possible with e.g. an embedding of the prompt provided as an input that stays fixed throughout the autoregressive step. Similar kind of idea, why a bit vector cannot be provided to disambiguate prompt from user tokens on input and output

Just in terms of doing inline data better, I think some models already train with "hidden" tokens that aren't exposed on input or output, but simply exist for delineation, so there can be no way to express the token in the user input unless the engine specifically inserts it

Re: Claude mixes up who said what

#148

Post by usernametaken29 »

adrian_b wrote: post #145
usernametaken29 wrote: post #131

Earlier quoted context omitted.

Actually at a hardware level floating point operations are not associative. So even with temperature of 0 you’re not mathematically guaranteed the same response. Hence, not deterministic.

You are right that as commonly implemented, the evaluation of an LLM may be non deterministic even when explicit randomization is eliminated, due to various race conditions in a concurrent evaluation. However, if you evaluate carefully the LLM core function, i.e. in a fixed order, you will obtain perfectly deterministic results (except on some consumer GPUs, where, due to memory overclocking, memory errors are freque…

Only that one is built to be deterministic and one is built to be probabilistic. Sure, you can technically force determinism but it is going to be very hard. Even just making sure your GPU is indeed doing what it should be doing is going to be hard. Much like debugging a CPU, but again, one is built for determinism and one is built for concurrency.

Re: Claude mixes up who said what

#149

Post by [deleted] »

satvikpendem wrote: post #93
idiotsecant wrote: post #89

Earlier quoted context omitted.

You don't think this is pedantry bordering on uselessness?

It's correcting a misconception that many people have regarding LLMs that they are inherently and fundamentally non-deterministic, as if they were a true random number generator, but they are closer to a pseudo random number generator in that they are deterministic with the right settings.

[deleted]

Re: Claude mixes up who said what

#150

Post by ehnto »

jwrallie wrote: post #49
lelandfe wrote: post #4

In chats that run long enough on ChatGPT, you'll see it begin to confuse prompts and responses, and eventually even confuse both for its system prompt . I suspect this sort of problem exists widely in AI.

I think it’s good to play with smaller models to have a grasp of these kind of problems, since they happen more often and are much less subtle.

Totally agree, these kinds of problems are really common in smaller models, and you build an intuition for when they're likely to happen.

The same issues are still happening in frontier models. Especially in long contexts or in the edges of the models training data.

Post reply on HN