Claude mixes up who said what • Hacker News Board

Re: Claude mixes up who said what

#71

Post by le-mark » Thu, Apr 09, 2026, 11:08 AM UTC

perching_aix wrote: post #47

sanitycheck wrote: post #32

Earlier quoted context omitted.

It's both, really. The companies selling us the service aren't saying "you should treat this LLM as a potentially hostile user on your machine and set up a new restricted account for it accordingly", they're just saying "download our app! connect it to all your stuff!" and we can't really blame ordinary users for doing that and getting into trouble.

There's a growing ecosystem of guardrailing methods, and these companies are contributing. Antrophic specifically puts in a lot of effort to better steer and characterize their models AFAIK. I primarily use Claude via VS Code, and it defaults to asking first before taking any action. It's simply not the wild west out here that you make it out to be, nor does it need to be. These are statistical systems, so issues can…

> It's simply not the wild west out here that you make it out to be

It is though. They are not talking about users using Claude code via vscode, they’re talking about non technical users creating apps that pipe user input to llms. This is a growing thing.

Re: Claude mixes up who said what

#72

Post by satvikpendem » Thu, Apr 09, 2026, 11:10 AM UTC

HPsquared wrote: post #69

hacker_homie wrote: post #64

Earlier quoted context omitted.

I have been saying this for a while, the issue is there's no good way to do LLM structured queries yet. There was an attempt to make a separate system prompt buffer, but it didn't work out and people want longer general contexts but I suspect we will end up back at something like this soon.

Fundamentally there's no way to deterministically guarantee anything about the output.

That is "fundamentally" not true, you can use a preset seed and temperature and get a deterministic output.

Re: Claude mixes up who said what

#73

Post by j-bos » Thu, Apr 09, 2026, 11:10 AM UTC

lelandfe wrote: post #4

In chats that run long enough on ChatGPT, you'll see it begin to confuse prompts and responses, and eventually even confuse both for its system prompt . I suspect this sort of problem exists widely in AI.

At work where LLM based tooling is being pushed haaard, I'm amazed every day that developers don't know, let alone second nature intuit, this and other emergent behavior of LLMs. But seeing that lack here on hn with an article on the frontpage boggles my mind. The future really is unevenly distributed.

Re: Claude mixes up who said what

#74

Post by throw310822 » Thu, Apr 09, 2026, 11:14 AM UTC

lelandfe wrote: post #4

In chats that run long enough on ChatGPT, you'll see it begin to confuse prompts and responses, and eventually even confuse both for its system prompt . I suspect this sort of problem exists widely in AI.

Makes me wonder if during training LLMs are asked to tell whether they've written something themselves or not. Should be quite easy: ask the LLM to produce many continuations of a prompt, then mix them with many other produced by humans, and then ask the LLM to tell them apart. This should be possible by introspecting on the hidden layers and comparing with the provided continuation. I believe Anthropic has already demonstrated that the models have already partially developed this capability, but should be trivial and useful to train it.

Re: Claude mixes up who said what

#75

Post by le-mark » Thu, Apr 09, 2026, 11:15 AM UTC

hacker_homie wrote: post #61

cookiengineer wrote: post #35

Earlier quoted context omitted.

Before 2023 I thought the way Star Trek portrayed humans fiddling with tech and not understanding any side effects was fiction. After 2023 I realized that's exactly how it's going to turn out. I just wish those self proclaimed AI engineers would go the extra mile and reimplement older models like RNNs, LSTMs, GRUs, DNCs and then go on to Transformers (or the Attention is all you need paper). This way they would under…

is this new tho, I don't know how to make a drill but I use them. I don't know how to make a car but i drive one. The issue I see is the personification, some people give vehicles names, and that's kinda ok because they usually don't talk back. I think like every technological leap people will learn to deal with LLMs, we have words like "hallucination" which really is the non personified version of lying. The next fe…

Do you not see your own contradiction? Cars and drills don’t kill people, self driving cars can! Normal cars can if they’re operated unsafely by human. These types of uncritical comments really highlight the level of euphoria in this moment.

Re: Claude mixes up who said what

#76

Post by ares623 » Thu, Apr 09, 2026, 11:18 AM UTC

nathell wrote: post #68

I’ve hit this! In my otherwise wildly successful attempt to translate a Haskell codebase to Clojure [0], Claude at one point asks: [Claude:] Shall I commit this progress? [some details about what has been accomplished follow] Then several background commands finish (by timeout or completing); Claude Code sees this as my input, thinks I haven’t replied to its question, so it answers itself in my name: [Claude:] Yes, g…

I wonder if tools like Terraform should remove the message "Run terraform apply plan.out next" that it prints after every `terraform plan` is run.

Re: Claude mixes up who said what

#77

Post by HPsquared » Thu, Apr 09, 2026, 11:19 AM UTC

satvikpendem wrote: post #72

HPsquared wrote: post #69

Earlier quoted context omitted.

Fundamentally there's no way to deterministically guarantee anything about the output.

That is "fundamentally" not true, you can use a preset seed and temperature and get a deterministic output.

I'll grant that you can guarantee the length of the output and, being a computer program, it's possible (though not always in practice) to rerun and get the same result each time, but that's not guaranteeing anything about said output.

Re: Claude mixes up who said what

#78

Post by 63stack » Thu, Apr 09, 2026, 11:20 AM UTC

They will roll out the "trusted agent platform sandbox" (I'm sure they will spend some time on a catchy name, like MythosGuard), and for only $19/month it will protect you from mistakes like throwing away your prod infra because the agent convinced itself that that is the right thing to do.

Of course MythosGuard won't be a complete solution either, but it will be just enough to steer the discourse into the "it's your own fault for running without MythosGuard really" area.

Re: Claude mixes up who said what

#79

Post by simianparrot » Thu, Apr 09, 2026, 11:21 AM UTC

satvikpendem wrote: post #72

HPsquared wrote: post #69

Earlier quoted context omitted.

Fundamentally there's no way to deterministically guarantee anything about the output.

That is "fundamentally" not true, you can use a preset seed and temperature and get a deterministic output.

A single byte change in the input changes the output. The sentence "Please do this for me" and "Please, do this for me" can lead to completely distinct output.

Given this, you can't treat it as deterministic even with temp 0 and fixed seed and no memory.

Re: Claude mixes up who said what

#80

Post by yunohn » Thu, Apr 09, 2026, 11:21 AM UTC

satvikpendem wrote: post #72

HPsquared wrote: post #69

Earlier quoted context omitted.

Fundamentally there's no way to deterministically guarantee anything about the output.

That is "fundamentally" not true, you can use a preset seed and temperature and get a deterministic output.

I initially thought the same, but apparently with the inaccuracies inherent to floating-point arithmetic and various other such accuracy leakage, it’s not true!

https://arxiv.org/html/2408.04667v5