Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Potential session/cache leakage between workspace instances or consumer accounts

github.com

Post reply on HN
Print view View original

111–120 of 151 posts

Re: Potential session/cache leakage between workspace instances or consumer accounts

#111

Post by cyral »

solenoid0937 wrote: post #96
pseudony wrote: post #85

Earlier quoted context omitted.

[flagged]

It's clear you haven't engaged with the subject matter beyond the typical "internet-forum cynic" mindset. Both companies were founded on the basis of AI Safety. - There are tons of great safety people doing real work at OpenAI. Releases are held back, models are evaluated, etc. - Anthropic goes even further - constrained themselves with a PBC/LTBT structure, treat safety even more rigorously, and notably delayed the…

To add to this, they should look at the Fable system card. It's 317 pages and it's clear how serious they are taking AI safety.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#112

Post by tadfisher »

andy99 wrote: post #61
paulddraper wrote: post #54

Earlier quoted context omitted.

Exactly. If you've never had an LLM (all models) suddenly start spouting nonsense in a completely different language...you haven't been using LLMs that much. They will go absolutely insane some % of the time.

Worth looking at https://www.anthropic.com/engineering/a-postmortem-of-three-... They can “go insane” but it seems often to be infra related as opposed to anything one would consider hallucination. Smaller models will often get stuck repeating a word or phrase forever but that’s a bit different and nobody would call it hallucination.

When you can reliably prompt these things into insanity, then it's demonstrably not an infrastructure issue.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#113

Post by pocksuppet »

albinowax_ wrote: post #90
markasoftware wrote: post #87

Earlier quoted context omitted.

Very true, this was likely an attack. Worth noting that mr kettle has done a defcon talk nearly every year on some variant of this attack, the most recent one titled "HTTP/1.1 must die" because he rightfully believes that switching to the binary headers of http/2 (specifically in reverse proxy connections to upstream servers) is the only way to systematically prevent these.

I’ll be back next month with a load of fresh vectors in “Can AI Do Novel Security Research? Meet the HTTP Terminator” https://portswigger.net/research/talks?talkId=36 Maybe my last presentation on the topic! Possibly.

Why the reference to AI? This looks like standard security research.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#114

Post by unknownfuture »

paulddraper wrote: post #54
Tiberium wrote: post #2

Sounds like a hallucination unless proven otherwise, even the leading LLMs can do those from time to time, and they will always appear plausible like that. Also could be the session having a lot previous context, like 800K+, which (I think) makes hallucinations more likely. Relevant comment from the OP which makes a hallucination more likely: > There is one tool call result that includes a string that printed a pathn…

Exactly. If you've never had an LLM (all models) suddenly start spouting nonsense in a completely different language...you haven't been using LLMs that much. They will go absolutely insane some % of the time.

I've used LLMs all day five days a week plus my own free time for the last year or so (new job).

I've seen plenty of hallucinations and context collapse behaviours.

I've never seen that.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#115

Post by jdw64 »

trq_ wrote: post #110

Hi, it's Thariq from the Claude Code Team here. Thanks for the detailed report. We’re confident this is a hallucination but of course take these reports seriously and the team is looking into it. We’ll report back if anything turns up.

I know it's the weekend, so thanks for working hard. Just a suggestion from a user: I wish we could manage Claude Code's memory more easily. Right now, when I go into the .claude folder and change a project folder name or something, sometimes it can't pull up the memory properly. It'd be nice if there were an easier way to import or export it. Thanks!

Re: Potential session/cache leakage between workspace instances or consumer accounts

#116

Post by Aurornis »

andy99 wrote: post #36
Tiberium wrote: post #2

Sounds like a hallucination unless proven otherwise, even the leading LLMs can do those from time to time, and they will always appear plausible like that. Also could be the session having a lot previous context, like 800K+, which (I think) makes hallucinations more likely. Relevant comment from the OP which makes a hallucination more likely: > There is one tool call result that includes a string that printed a pathn…

I realize hallucination has no precise definition but this doesn’t sound at all like anything I’ve ever heard called hallucination. Hallucination is usually plausible wrong answers or made up info that ends up fitting the most likely response (like a manufactured citation) and comes from the way LLMs work at predicting tokens. This example demonstrates completely implausible output, it’s not something that fits with…

The word “hallucination” has become overloaded, but it general means an LLM producing some output that isn’t plausible or grounded. When you have a very long context session where the context includes “minecraft.py” it’s not hard to extrapolate that Minecraft may have ended up in one of the reasoning traces and that distraction snowballed until it appeared in the output.

These effects are becoming more rare as the SOTA models are improving so much. If you spent a lot of time with earlier LLMs or you experiment with smaller, quantized local LLM models this type of thing happens very frequently. When you see it happen so much on a model you’re running on your own hardware it becomes a reflex to chuckle and reset the session with a clean context. When it happens from a hosted provider it can be scarier because it’s not the type of failure mode most people are used to seeing.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#118

Post by Aurornis »

bix6 wrote: post #20

So the options are this amazing tech is so stupid it just randomly brings up Minecraft or it’s got a major security issue?

The person had “minecraft.py” in their context and the session context was very long.

Having an LLM session with very long context occasionally go off on a tangent is not uncommon. The people who expect absolute perfection out of every LLM interaction see this as some total indictment of the entire technology, but the people who use these tools daily have learned to treat the output as partially stochastic and to avoid extremely long context, even if the model offers it. It’s best to compact strategically or summarize next steps to hand off to a new session. Using sub-sessions can also reduce context pollution at the cost of additional token expenditure to summarize and transfer data to and from the sub-session.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#119

Post by superb_dev »

cyral wrote: post #111
solenoid0937 wrote: post #96

Earlier quoted context omitted.

It's clear you haven't engaged with the subject matter beyond the typical "internet-forum cynic" mindset. Both companies were founded on the basis of AI Safety. - There are tons of great safety people doing real work at OpenAI. Releases are held back, models are evaluated, etc. - Anthropic goes even further - constrained themselves with a PBC/LTBT structure, treat safety even more rigorously, and notably delayed the…

To add to this, they should look at the Fable system card. It's 317 pages and it's clear how serious they are taking AI safety.

Page count is not a measure of how seriously something is being taken when you can easily generate pages and pages of slop

Re: Potential session/cache leakage between workspace instances or consumer accounts

#120

Post by andy99 »

tadfisher wrote: post #112
andy99 wrote: post #61

Earlier quoted context omitted.

Worth looking at https://www.anthropic.com/engineering/a-postmortem-of-three-... They can “go insane” but it seems often to be infra related as opposed to anything one would consider hallucination. Smaller models will often get stuck repeating a word or phrase forever but that’s a bit different and nobody would call it hallucination.

When you can reliably prompt these things into insanity, then it's demonstrably not an infrastructure issue.

Can you explain that please?

(Not the syllogism, the premise)

Post reply on HN