Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

231–240 of 378 posts

Re: Claude mixes up who said what

#231

Post by VikingCoder »

fzeindl wrote: post #104
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

The "S" in "LLM" is for "Security".

Re: Claude mixes up who said what

#232

Post by jerf »

orbital-decay wrote: post #211

Claude in particular has nothing to do with it. I see many people are discovering the well-known fundamental biases and phenomena in LLMs again and again. There are many of those. The best intuition is treating the context as "kind of but not quite" an associative memory, instead of a sequence or a text file with tokens. This is vaguely similar to what humans are good and bad at, and makes it obvious what is easy and…

You can really see this in the recent video generation where they try to incorporate text-to-speech into the video. All the tokens flying around, all the video data, all the context of all human knowledge ever put into bytes ingested into it, and the systems still completely routinely (from what I can tell) fails to put the speech in the right mouth even with explicit instruction and all the "common sense" making it obvious who is saying what.

There was some chatter yesterday on HN about the very strange capability frontier these models have and this is one of the biggest ones I can think of... a model that de novo, from scratch is generating megabyte upon megabyte of really quite good video information that at the same time is often unclear on the idea that a knock-knock joke does not start with the exact same person saying "Knock knock? Who's there?" in one utterance.

Re: Claude mixes up who said what

#234

Post by ikidd »

xigoi wrote: post #198
hacker_homie wrote: post #64

Earlier quoted context omitted.

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

How long is it going to take before vibe coders reinvent normal programming?

I'd like to share my project that let's you hit Tab in order to get a list of possible methods/properties for your defined object, then actually choose a method or property to complete the object string in code.

I wrote it in Typescript and React.

Please star on Github.

Re: Claude mixes up who said what

#235

Post by WarmWash »

benhurmarcel wrote: post #223
insin wrote: post #20

Earlier quoted context omitted.

Gemini seems to be an expert in mistaking its own terrible suggestions as written by you, if you keep going instead of pruning the context

In Gemini chat I find that you should avoid continuing a conversation if its answer was wrong or had a big shortcoming. It's better to edit the previous prompt so that it comes up with a better answer in the first place, instead of sending a new message.

The key with gemini is to migrate to a new chat once it makes a single dumb mistake. It's a very strong model, but once it steps in the mud, you'll lose your mind trying to recover it.

Delete the bad response, ask it for a summary or to update [context].md, then start a new instance.

Re: Claude mixes up who said what

#236

Post by vidarh »

kg wrote: post #227
vidarh wrote: post #217

Earlier quoted context omitted.

This makes no sense to me. Being fooled into thinking data is instruction is exactly evidence of an inability to reliably distinguish them. And being coerced or convinced to bypass rules is exactly what prompt injection is, and very much not uniquely human any more.

The email from your boss and the email from a sender masquerading as your boss are both coming through the same channel in the same format with the same presentation, which is why the attack works. Unless you were both faceblind and bad at recognizing voices, the same attack wouldn't work in-person, you'd know the attacker wasn't your boss. Many defense mechanisms used in corporate email environments are built around…

> The email from your boss and the email from a sender masquerading as your boss are both coming through the same channel in the same format with the same presentation, which is why the attack works.

Yes, that is exactly the point.

> Unless you were both faceblind and bad at recognizing voices, the same attack wouldn't work in-person, you'd know the attacker wasn't your boss.

Irrelevant, as other attacks works then. E.g. it is never a given that your bosses instructions are consistent with the terms of your employment, for example.

> Prompt injection is just exploiting the lack of separation, it's not 'coercion' or 'convincing'. Though you could argue that things like jailbreaking are closer to coercion, I'm not convinced that a statistical token predictor can be coerced to do anything.

It is very much "convincing", yes. The ability to convince an LLM is what creates the effective lack of separation. Without that, just using "magic" values and a system prompt telling it to ignore everything inside would create separation. But because text anywhere in context can convince the LLM to disregard previous rules, there is no separation.

Re: Claude mixes up who said what

#237

Post by novaleaf »

in Claude Code's conversation transcripts it stores messages from subagents as type="user". I always thought this was odd, and I guess this is the consequence of going all-in on vibing.

There are some other metafields like isSidechain=true and/or type="tool_result" that are technically enough to distinguish actual user vs subagent messages, though evidently not enough of a hint for claude itself.

Source: I'm writing a wrapper for Claude Code so am dealing with this stuff directly.

Re: Claude mixes up who said what

#238

Post by pixl97 »

adrian_b wrote: post #145
usernametaken29 wrote: post #131

Earlier quoted context omitted.

Actually at a hardware level floating point operations are not associative. So even with temperature of 0 you’re not mathematically guaranteed the same response. Hence, not deterministic.

You are right that as commonly implemented, the evaluation of an LLM may be non deterministic even when explicit randomization is eliminated, due to various race conditions in a concurrent evaluation. However, if you evaluate carefully the LLM core function, i.e. in a fixed order, you will obtain perfectly deterministic results (except on some consumer GPUs, where, due to memory overclocking, memory errors are freque…

If you want a deterministic LLM, just build 'Plain old software'.

Re: Claude mixes up who said what

#239

Post by orbital-decay »

codebje wrote: post #170
mt_ wrote: post #152

Earlier quoted context omitted.

Exactly like human input to output.

Well no, nothing like that, because customers and bosses are clearly different forms of interaction.

These are different "agents" in LLM terms, they have separate contexts and separate training

Re: Claude mixes up who said what

#240

Post by pixl97 »

timcobb wrote: post #124
HeavyStorm wrote: post #102

Earlier quoted context omitted.

The real issue is expecting an LLM to be deterministic when it's not.

they are deterministic, open a dev console and run the same prompt two times w/ temperature = 0

And then the 3rd time it shows up differently leaving you puzzled on why that happened.

The deterministic has a lot of 'terms and conditions' apply depending on how it's executing on the underlying hardware.

Post reply on HN