Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Potential session/cache leakage between workspace instances or consumer accounts

github.com

Post reply on HN
Print view View original

11–20 of 151 posts

Re: Potential session/cache leakage between workspace instances or consumer accounts

#11

Post by macNchz »

ec109685 wrote: post #8

Caching doesn’t work the way the bug reporter implies. Caches are shared (at least across the enterprise), but its key is always a function of the input before it. We achieved significant savings simply by moving everything that varies across individuals out of the system prompt so every session starts from a cache point. For example you never want your system prompt to start with the time that the session started. M…

Caching is not supposed to work like that, but that doesn’t preclude the cache key computation function from having bugs.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#12

Post by supriyo-biswas »

ec109685 wrote: post #8

Caching doesn’t work the way the bug reporter implies. Caches are shared (at least across the enterprise), but its key is always a function of the input before it. We achieved significant savings simply by moving everything that varies across individuals out of the system prompt so every session starts from a cache point. For example you never want your system prompt to start with the time that the session started. M…

There could just also be a bug where the output tokens of session 1 were shared with session 2, due to a race condition or similar.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#14

Post by macNchz »

Tiberium wrote: post #2

Sounds like a hallucination unless proven otherwise, even the leading LLMs can do those from time to time, and they will always appear plausible like that. Also could be the session having a lot previous context, like 800K+, which (I think) makes hallucinations more likely. Relevant comment from the OP which makes a hallucination more likely: > There is one tool call result that includes a string that printed a pathn…

The person posting this claims to have reproduced in a separate context down the thread:

> Same thing just happened on a Claude Mobile session in same Enterprise account. Common theme in both is Sonnet 5, first response after more than 5 minutes (cache miss).

Re: Potential session/cache leakage between workspace instances or consumer accounts

#15

Post by marginalia_nu »

macNchz wrote: post #11
ec109685 wrote: post #8

Caching doesn’t work the way the bug reporter implies. Caches are shared (at least across the enterprise), but its key is always a function of the input before it. We achieved significant savings simply by moving everything that varies across individuals out of the system prompt so every session starts from a cache point. For example you never want your system prompt to start with the time that the session started. M…

Caching is not supposed to work like that, but that doesn’t preclude the cache key computation function from having bugs.

Yeah there's quite a lot of potential bugs that could have this shape. If I were to guess it could be a buffer in a buffer pool not being sized and zeroed correctly, allowing stale data to bleed between sessions.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#16

Post by ethagnawl »

consp wrote: post #10
emehex wrote: post #5

Earlier quoted context omitted.

"Coding is largely solved"

While abused by LLM vendors, that phrase in one form or another I've been hearing since the early '00s and it's likely way older.

Sure but have you ever seen it actually play out in practice like it currently is? Whether or not it's true (of course it's not) people are currently behaving as if it is and firing/hiring accordingly.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#17

Post by 27183 »

jstummbillig wrote: post #13

Is there anything particular about LLMs that would make separating customer data harder than in all SaaS cases?

If I had to hazard a guess, doing anything in a multi-tenant way on a GPU is going to be hard mode compared to most SaaS due to lack of memory safe tooling. I've built multi-tenant SaaS systems, and I've done a little GPU programming (a long time ago), but I've never tried to combine the two disciplines.

Re: Potential session/cache leakage between workspace instances or consumer accounts

#19

Post by woadwarrior01 »

jstummbillig wrote: post #13

Is there anything particular about LLMs that would make separating customer data harder than in all SaaS cases?

It'd be terribly compute inefficient to not share prefix caches (KV cache) across customers.
Post reply on HN