Claude mixes up who said what
dwyer.co.za
Claude mixes up who said what
1–10 of 378 posts
Re: Claude mixes up who said what
#2[deleted]
Re: Claude mixes up who said what
#3terrifying. not in any "ai takes over the world" sense but more in the sense that this class of bug lets it agree with itself which is always where the worst behavior of agents comes from.
Re: Claude mixes up who said what
#4In chats that run long enough on ChatGPT, you'll see it begin to confuse prompts and responses, and eventually even confuse both for its system prompt. I suspect this sort of problem exists widely in AI.
Re: Claude mixes up who said what
#5Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees.
It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted the second you put any user input into the prompt.
Re: Claude mixes up who said what
#6The statement that current AI are "juniors" that need to be checked and managed still holds true. It is a tool based on probabilities.
If you are fine with giving every keys and write accesses to your junior because you think they will probability do the correct thing and make no mistake, then it's on you.
Like with juniors, you can vent on online forums, but ultimately you removed all the fool's guard you got and what they did has been done.
Re: Claude mixes up who said what
#7What do you mean that's not OK?
It's "AGI" because humans do it too and we mix up names and who said what as well. /s
Re: Claude mixes up who said what
#8Why are tokens not coloured? Would there just be too many params if we double the token count so the model could always tell input tokens from output tokens?
Re: Claude mixes up who said what
#9The statement that current AI are "juniors" that need to be checked and managed still holds true. It is a tool based on probabilities . If you are fine with giving every keys and write accesses to your junior because you think they will probability do the correct thing and make no mistake, then it's on you. Like with juniors, you can vent on online forums, but ultimately you removed all the fool's guard you got and w…
> If you are fine with giving every keys and write accesses to your junior because you think they will probability do the correct thing and make no mistake, then it's on you.
How is that different from a senior?
Re: Claude mixes up who said what
#10one of my favourite genres of AI generated content is when someone gets so mad at Claude they order it to make a massive self-flagellatory artefact letting the world know how much it sucks