So the options are this amazing tech is so stupid it just randomly brings up Minecraft or it’s got a major security issue?
¿Por qué no los dos?
Potential session/cache leakage between workspace instances or consumer accounts
21–30 of 151 posts
Re: Potential session/cache leakage between workspace instances or consumer accounts
#22Is there anything particular about LLMs that would make separating customer data harder than in all SaaS cases?
It'd be terribly compute inefficient to not share prefix caches (KV cache) across customers.
What is the probability that two customers will have exactly the same tokens in cache? Wouldnt it require using the exact same CLAUDE.md, skills, MCPs and context? After that it is even worse since the nondeterminism of LLMs and humans
Re: Potential session/cache leakage between workspace instances or consumer accounts
#23[deleted]
Re: Potential session/cache leakage between workspace instances or consumer accounts
#24happy fourth of july everybody!
Re: Potential session/cache leakage between workspace instances or consumer accounts
#25Caching doesn’t work the way the bug reporter implies. Caches are shared (at least across the enterprise), but its key is always a function of the input before it. We achieved significant savings simply by moving everything that varies across individuals out of the system prompt so every session starts from a cache point. For example you never want your system prompt to start with the time that the session started. M…
There is a massive incentive for optimization, so I expect they’re doing a ton of very clever tricks, all of which make this kind of bug more likely.
Re: Potential session/cache leakage between workspace instances or consumer accounts
#26Sounds like a hallucination unless proven otherwise, even the leading LLMs can do those from time to time, and they will always appear plausible like that. Also could be the session having a lot previous context, like 800K+, which (I think) makes hallucinations more likely. Relevant comment from the OP which makes a hallucination more likely: > There is one tool call result that includes a string that printed a pathn…
Why? what does make it more likely?
Re: Potential session/cache leakage between workspace instances or consumer accounts
#27Is there anything particular about LLMs that would make separating customer data harder than in all SaaS cases?
Vibe-coding the implementation.
I haven't had much issue with Codex, but seems Claude Code has major issues being reported nearly on the daily.
They also happen to be the most boastful about not reading or looking at the code.
LLMs are very capable, but not nearly to the level they seem to be messaging.
(We've actually moved on from vibe-coding to having the LLM vibe code itself in a loop)
Re: Potential session/cache leakage between workspace instances or consumer accounts
#28Earlier quoted context omitted.
It'd be terribly compute inefficient to not share prefix caches (KV cache) across customers.
What is the probability that two customers will have exactly the same tokens in cache? Wouldnt it require using the exact same CLAUDE.md, skills, MCPs and context? After that it is even worse since the nondeterminism of LLMs and humans
I suspect what GP is getting at is there will be a strong incentive to implement some structural sharing across tenants to avoid redundantly storing the same tokens over and over. At least I'd be tempted to do this if I was working with a very precious, constrained resource (e.g. VRAM). Doing this correctly seems.. very difficult. [edit] To answer your question directly: the probability that the entire cache is identical between two different users is very low, but the probability that there exists identical chunks of cache between two different users is very high. Exploiting those commonalities successfully will significantly compress the data.
Re: Potential session/cache leakage between workspace instances or consumer accounts
#29Note the repro condition: first response after 5+ min, i.e. a cache miss. A cache leak would show up on hits (someone else's cached prefix), not on misses where everything is recomputed from your own tokens.
Re: Potential session/cache leakage between workspace instances or consumer accounts
#30Caching doesn’t work the way the bug reporter implies. Caches are shared (at least across the enterprise), but its key is always a function of the input before it. We achieved significant savings simply by moving everything that varies across individuals out of the system prompt so every session starts from a cache point. For example you never want your system prompt to start with the time that the session started. M…
Hash functions necesarily have collisions. Also, it is perfectly possible to introduce bugs in the hash function (hash inputs, hash function itself) that allows cross account contamination.