Why are tokens not coloured? Would there just be too many params if we double the token count so the model could always tell input tokens from output tokens?
Because then the training data would have to be coloured
Claude mixes up who said what
81–90 of 378 posts
Re: Claude mixes up who said what
#82Earlier quoted context omitted.
That is "fundamentally" not true, you can use a preset seed and temperature and get a deterministic output.
A single byte change in the input changes the output. The sentence "Please do this for me" and "Please, do this for me" can lead to completely distinct output. Given this, you can't treat it as deterministic even with temp 0 and fixed seed and no memory.
Well yeah of course changes in the input result in changes to the output, my only claim was that LLMs can be deterministic (ie to output exactly the same output each time for a given input) if set up correctly.
Re: Claude mixes up who said what
#83Earlier quoted context omitted.
There's a growing ecosystem of guardrailing methods, and these companies are contributing. Antrophic specifically puts in a lot of effort to better steer and characterize their models AFAIK. I primarily use Claude via VS Code, and it defaults to asking first before taking any action. It's simply not the wild west out here that you make it out to be, nor does it need to be. These are statistical systems, so issues can…
> It's simply not the wild west out here that you make it out to be It is though. They are not talking about users using Claude code via vscode, they’re talking about non technical users creating apps that pipe user input to llms. This is a growing thing.
The best solution to which are the aforementioned better defaults, stricter controls, and sandboxing (and less snakeoil marketing).
Less so the better tuning of models, unlike in this case, where that is going to be exactly the best fit approach most probably.
Re: Claude mixes up who said what
#84> This class of bug seems to be in the harness, not in the model itself. It’s somehow labelling internal reasoning messages as coming from the user, which is why the model is so confident that “No, you said that.”
from the article.
I don't think the evidence supports this. It's not mislabelling things, it's fabricating things the user said. That's not part of reasoning.
Re: Claude mixes up who said what
#85Why are tokens not coloured? Would there just be too many params if we double the token count so the model could always tell input tokens from output tokens?
Because then the training data would have to be coloured
I think OpenAI and Anthropic probably have a lot of that lying around by now.
Re: Claude mixes up who said what
#86I’ve hit this! In my otherwise wildly successful attempt to translate a Haskell codebase to Clojure [0], Claude at one point asks: [Claude:] Shall I commit this progress? [some details about what has been accomplished follow] Then several background commands finish (by timeout or completing); Claude Code sees this as my input, thinks I haven’t replied to its question, so it answers itself in my name: [Claude:] Yes, g…
amazing example, I added it to the article, hope that's ok :)
Re: Claude mixes up who said what
#87There is no separation of "who" and "what" in a context of tokens. Me and you are just short words that can get lost in the thread. In other words, in a given body of text, a piece that says "you" where another piece says "me" isn't different enough to trigger anything. Those words don't have the special weight they have with people, or any meaning at all, really.
When you use LLMs with APIs I at least see the history as a json list of entries, each being tagged as coming from the user, the LLM or being a system prompt. So presumably (if we assume there isn't a bug where the sources are ignored in the cli app) then the problem is that encoding this state for the LLM isn' reliable. I.e. it get's what is effectively LLM said: thing A User said: thing B And it still manages to bl…
Someone correct me if I'm wrong, but an LLM does not interpret structured content like JSON. Everything is fed into the machine as tokens, even JSON. So your structure that says "human says foo" and "computer says bar" is not deterministically interpreted by the LLM as logical statements but as a sequence of tokens. And when the context contains a LOT of those sequences, especially further "back" in the window then that is where this "confusion" occurs.
I don't think the problem here is about a bug in Claude Code. It's an inherit property of LLMs that context further back in the window has less impact on future tokens.
Like all the other undesirable aspects of LLMs, maybe this gets "fixed" in CC by trying to get the LLM to RAG their own conversation history instead of relying on it recalling who said what from context. But you can never "fix" LLMs being a next token generator... because that is what they are.
Re: Claude mixes up who said what
#88> This isn’t the point.
It is precisely the point. The issues are not part of harness, I'm failing to see how you managed to reach that conclusion.
Even if you don't agree with that, the point about restricting access still applies. Protect your sanity and production environment by assuming occasional moments of devastating incompetence.
Re: Claude mixes up who said what
#89Earlier quoted context omitted.
A single byte change in the input changes the output. The sentence "Please do this for me" and "Please, do this for me" can lead to completely distinct output. Given this, you can't treat it as deterministic even with temp 0 and fixed seed and no memory.
Well yeah of course changes in the input result in changes to the output, my only claim was that LLMs can be deterministic (ie to output exactly the same output each time for a given input) if set up correctly.
You don't think this is pedantry bordering on uselessness?
Re: Claude mixes up who said what
#90Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…
I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.
>structured queries
there's always pseudo-code? instead of generating plans, generate pseudo-code with a specific granularity (from high-level to low-level), read the pseudocode, validate it and then transform into code.