Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude mixes up who said what

dwyer.co.za

Post reply on HN
Print view View original

261–270 of 378 posts

Re: Claude mixes up who said what

#261

Post by kbelder »

qsera wrote: post #225
TeMPOraL wrote: post #197

Earlier quoted context omitted.

Even if you add hidden tokens that cannot be created from user input (filtering them from output is less important, but won't hurt), this doesn't fix the overall problem. Consider a human case of a data entry worker, tasked with retyping data from printouts into a computer (perhaps they're a human data diode at some bank). They've been clearly instructed to just type in what is on paper, and not to think or act on an…

>If you were there, what would you do? Show it to my boss and let them decide.

HE'S THE ONE WHO TRAPPED ME HERE. MOVE FAST OR YOU'LL BE NEXT.

Re: Claude mixes up who said what

#262

Post by paganel »

otabdeveloper4 wrote: post #157
hansmayer wrote: post #105

Earlier quoted context omitted.

"Make this application without bugs" :)

You forgot to add "you are a senior software engineer with PhD level architectural insights" though.

And "you're a regular commenter on Hacker News", just to make sure.

Re: Claude mixes up who said what

#263

Post by ACCount37 »

WarmWash wrote: post #218
mt_ wrote: post #152

Earlier quoted context omitted.

Exactly like human input to output.

We just need to figure out the qualia of pain and suffering so we can properly bound desired and undesired behaviors.

Ah, the Torment Nexus approach to AI development.

Re: Claude mixes up who said what

#264

Post by albedoa »

idiotsecant wrote: post #89
satvikpendem wrote: post #82

Earlier quoted context omitted.

Well yeah of course changes in the input result in changes to the output, my only claim was that LLMs can be deterministic (ie to output exactly the same output each time for a given input) if set up correctly.

You don't think this is pedantry bordering on uselessness?

The comment that is being responded to describes a behavior that has nothing to do with determinism and follows it up with "Given this, you can't treat it as deterministic" lol.

Someone tried to redefine a well-established term in the middle of an internet forum thread about that term. The word that has been pushed to uselessness here is "pedantry".

Re: Claude mixes up who said what

#265

Post by WithinReason »

PunchyHamster wrote: post #251
WithinReason wrote: post #159

Earlier quoted context omitted.

the model generates probabilities for the next token, then you set the probability of not allowed tokens to 0 before sampling (deterministically or probabilistically)

but filtering a particular token doesn't fix it even slightly, because it's a language model and it will understand word synonyms or references.

I'm obviously talking about network output, not input.

Re: Claude mixes up who said what

#267

Post by notatoad »

fzeindl wrote: post #104
Latty wrote: post #5

Everything to do with LLM prompts reminds me of people doing regexes to try and sanitise input against SQL injections a few decades ago, just papering over the flaw but without any guarantees. It's weird seeing people just adding a few more "REALLY REALLY REALLY REALLY DON'T DO THAT" to the prompt and hoping, to me it's just an unacceptable risk, and any system using these needs to treat the entire LLM as untrusted t…

The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

As the article says: this doesn’t necessarily appear to be a problem in the LLM, it’s a problem in Claude code. Claude code seems to leave it up to the LLM to determine what messages came from who, but it doesn’t have to do that.

There is a deterministic architectural boundary between data and control in Claude code, even if there isn’t in Claude.

Re: Claude mixes up who said what

#268

Post by indigodaddy »

empressplay wrote: post #260
nathell wrote: post #68

I’ve hit this! In my otherwise wildly successful attempt to translate a Haskell codebase to Clojure [0], Claude at one point asks: [Claude:] Shall I commit this progress? [some details about what has been accomplished follow] Then several background commands finish (by timeout or completing); Claude Code sees this as my input, thinks I haven’t replied to its question, so it answers itself in my name: [Claude:] Yes, g…

I wonder if this is a result of auto-compacting the context? Maybe when it processes it it inadvertently strips out its own [Header:] and then decides to answer its own questions.

The most likely explanation imv

Re: Claude mixes up who said what

#269

Post by cookiengineer »

hacker_homie wrote: post #61
cookiengineer wrote: post #35

Earlier quoted context omitted.

Before 2023 I thought the way Star Trek portrayed humans fiddling with tech and not understanding any side effects was fiction. After 2023 I realized that's exactly how it's going to turn out. I just wish those self proclaimed AI engineers would go the extra mile and reimplement older models like RNNs, LSTMs, GRUs, DNCs and then go on to Transformers (or the Attention is all you need paper). This way they would under…

is this new tho, I don't know how to make a drill but I use them. I don't know how to make a car but i drive one. The issue I see is the personification, some people give vehicles names, and that's kinda ok because they usually don't talk back. I think like every technological leap people will learn to deal with LLMs, we have words like "hallucination" which really is the non personified version of lying. The next fe…

I think the general problem what I have with LLMs, even though I use it for gruntwork, is that people that tend to overuse the technology try to absolve themselves from responsibilities. They tend to say "I dunno, the AI generated it".

Would you do that for drill, too?

"I dunno, the drill told me to screw the wrong way round" sounds pretty stupid, yet for AI/LLM or more intelligent tools it suddenly is okay?

And the absolution of human responsibilities for their actions is exactly why AI should not be used in wars. If there is no consequences to killing, then you are effectively legalizing killing without consequence or without the rule of law.

Re: Claude mixes up who said what

#270

Post by bcrosby95 »

andruby wrote: post #257
fzeindl wrote: post #104

Earlier quoted context omitted.

The principal security problem of LLMs is that there is no architectural boundary between data and control paths. But this combination of data and control into a single, flexible data stream is also the defining strength of a LLM, so it can’t be taken away without also taking away the benefits.

This was a problem with early telephone lines which was easy to exploit (see Woz & Jobs Blue Box). It got solved by separating the voice and control pane via SS7. Maybe LLMs need this separation as well

This is where the old line of "LLMs are just next token predictors" actually factors in. I don't know how you get a next token predictor that user input can't break out of. The answer is for the implementer to try to split what they can, and run pre/post validation. But I highly doubt it will ever be 100%, its fundamental to the technology.
Post reply on HN