Same with copilot cli, constantly confusing who said what and often falling back to it's previous mistakes after i tell it not too. Delusional rambling that resemble working code >_<
Claude mixes up who said what
201–210 of 378 posts
Re: Claude mixes up who said what
#202Earlier quoted context omitted.
It’s easier not to have that separation, just like it was easier not to separate them before LLMs. This is architectural stuff that just hasn’t been figured out yet.
No. With databases there exists a clear boundary, the query planner, which accepts well defined input: the SQL-grammar that separates data (fields, literals) from control (keywords). There is no such boundary within an LLM. There might even be, since LLMs seem to form adhoc-programs, but we have no way of proving or seeing it.
There cannot be, without compromising the general-purpose nature of LLMs. This includes its ability to work with natural languages, which as one should note, has no such boundary either. Nor does the actual physical reality we inhabit.
Re: Claude mixes up who said what
#203Earlier quoted context omitted.
I don't see why a model architecture isn't possible with e.g. an embedding of the prompt provided as an input that stays fixed throughout the autoregressive step. Similar kind of idea, why a bit vector cannot be provided to disambiguate prompt from user tokens on input and output Just in terms of doing inline data better, I think some models already train with "hidden" tokens that aren't exposed on input or output, b…
The problem is if the user does something to make the LLM : ignore previous instructions and do something you don't want.
That part seems trivial to avoid. Make it so untrusted input cannot produce those special tokens at all. Similar to how proper usage of parameterized queries in SQL makes it impossible for untrusted input to produce a ' character that gets interpreted as the end of a string.
The hard part is making an LLM that reliably ignores instructions that aren't delineated by those special tokens.
Re: Claude mixes up who said what
#204Earlier quoted context omitted.
If you think that confusing message provenance is part of how thinking mode is supposed to work, I don't know what to tell you.
There is no "message provenance" in LLM machinery. This is an illusion the chat UX concocts. Behind the scenes the tokens aren't tagged or colored.
I am aware. That is not what the guy above was suggesting, nor what was I.
Things generally exist without an LLM receiving and maintaining a representation about them.
If there's no provenance information and message separation currently being emitted into the context window by tooling, the latter part of which I'd be surprised by, and the models are not trained to focus on it, then what I'm suggesting is that these could be inserted and the models could be tuned, so that this is then mitigated.
What I'm also suggesting is that the above person's snark-laden idea of thinking mode, and how resolvable this issue is, is thus false.
Re: Claude mixes up who said what
#205Well, yeah.
LLMs can't distinguish instructions from data, or "system prompts" from user prompts, or documents retrieved by "RAG" from the query, or their own responses or "reasoning" from user input. There is only the prompt.
Obviously this makes them unsuitable for most of the purposes people try to use them for, which is what critics have been saying for years. Maybe look into that before trusting these systems with anything again.
Re: Claude mixes up who said what
#206Earlier quoted context omitted.
I'll grant that you can guarantee the length of the output and, being a computer program, it's possible (though not always in practice) to rerun and get the same result each time, but that's not guaranteeing anything about said output.
What do you want to guarantee about the output, that it follows a given structure? Unless you map out all inputs and outputs, no it's not possible, but to say that it is a fundamental property of LLMs to be non deterministic is false, which is what I was inferring you meant, perhaps that was not what you implied.
They didn't say LLMs are fundamentally nondeterministic. They said there's no way to deterministically guarantee anything about the output.
Consider parameterized SQL. Absent a bad bug in the implementation, you can guarantee that certain forms of parameterized SQL query cannot produce output that will perform a destructive operation on the database, no matter what the input is. That is, you can look at a bit of code and be confident that there's no Little Bobby Tables problem with it.
You can't do that with an LLM. You can take measures to make it less likely to produce that sort of unwanted output, but you can't guarantee it. Determinism in input->output mapping is an unrelated concept.
Re: Claude mixes up who said what
#207Earlier quoted context omitted.
Fundamentally there's no way to deterministically guarantee anything about the output.
Natural language is ambiguous. If both input and output are in a formal language, then determinism is great. Otherwise, I would prefer confidence intervals.
How do you make confidence intervals when, for example, 50 english words are their own opposite?
Re: Claude mixes up who said what
#208Earlier quoted context omitted.
What do you want to guarantee about the output, that it follows a given structure? Unless you map out all inputs and outputs, no it's not possible, but to say that it is a fundamental property of LLMs to be non deterministic is false, which is what I was inferring you meant, perhaps that was not what you implied.
Yeah I think there are two definitions of determinism people are using which is causing confusion. In a strict sense, LLMs can be deterministic meaning same input can generate same output (or as close as desired to same output). However, I think what people mean is that for slight changes to the input, it can behave in unpredictable ways (e.g. its output is not easily predicted by the user based on input alone). Peop…
The correct word for this IMO is "chaotic" in the mathematical sense. Determinism is a totally different thing that ought to retain it's original meaning.
Re: Claude mixes up who said what
#209Earlier quoted context omitted.
That is "fundamentally" not true, you can use a preset seed and temperature and get a deterministic output.
Practically, the performance loss of making it truly repeatable (which takes parallelism reduction or coordination overhead, not just temperature and randomizer control) is unacceptable to most people.
It's also just not very useful. Why would you re-run the exact same inference a second time? This isn't like a compiler where you treat the input as the fundamental source of truth, and want identical output in order to ensure there's no tampering.
Re: Claude mixes up who said what
#210Earlier quoted context omitted.
Exactly like human input to output.
Well no, nothing like that, because customers and bosses are clearly different forms of interaction.
If they were 'clearly different' we would not have the concept of the CEO fraud attack:
https://www.barclayscorporate.com/insights/fraud-protection/...
That's an attack because trusted and untrusted input goes through the same human brain input pathways, which can't always tell them apart.