Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt caching for cheaper LLM tokens

ngrok.com

Post reply on HN
Print view View original

61–70 of 76 posts

Re: Prompt caching for cheaper LLM tokens

#61

Post by IanCal »

gwern wrote: post #60
samwho wrote: post #33

Earlier quoted context omitted.

The only thing that comes to mind is some kind of timing attack. Send loads of requests specific to a company you’re trying to spy on and if it comes back cached you know someone has sent that prompt recently. Expensive attack, though, with a large search space.

No, the search space is tiny: you can just attack 1 BPE at a time! Stuff like password guessing is almost trivial when you get to do a timing attack on each successive character. So that lets you quickly exfiltrate arbitrary numbers of prompts, especially if you have any idea what you are looking for. (Note that a lot of prompts are already public information, or you can already exfiltrate prompts quite easily from s…

Do any providers do this level of granularity? Anthropic require explicit cache markers, for example.

Re: Prompt caching for cheaper LLM tokens

#62

Post by IanCal »

moebrowne wrote: post #31
samwho wrote: post #27

Earlier quoted context omitted.

With KV caching as it’s described there it has to be a prefix match. OpenAI state in their docs they don’t cache anything below 1024 tokens long, and I’m sure I read somewhere that they only cache in 1024 token blocks (so 1024, 2048, 3072, etc) but I can’t find it now. There’s been some research into how to cache chunks in the middle, but I don’t think any of the providers are doing it yet because it needs the prompt…

https://platform.openai.com/docs/guides/prompt-caching#requi... > Caching is available for prompts containing 1024 tokens or more. No mention of caching being in blocks of 1024 tokens thereafter.

At launch it was described as being in blocks of 128

https://openai.com/index/api-prompt-caching/

Re: Prompt caching for cheaper LLM tokens

#63

Post by reitzensteinm »

gwern wrote: post #60
samwho wrote: post #33

Earlier quoted context omitted.

The only thing that comes to mind is some kind of timing attack. Send loads of requests specific to a company you’re trying to spy on and if it comes back cached you know someone has sent that prompt recently. Expensive attack, though, with a large search space.

No, the search space is tiny: you can just attack 1 BPE at a time! Stuff like password guessing is almost trivial when you get to do a timing attack on each successive character. So that lets you quickly exfiltrate arbitrary numbers of prompts, especially if you have any idea what you are looking for. (Note that a lot of prompts are already public information, or you can already exfiltrate prompts quite easily from s…

Hill climbing a password would only be possible if intermediate KV cache entries were stored. To hillclimb "hunter2", you're going to try "a", "b", "c", etc, until you notice that "h" comes back faster. Then you try "ha", "hb" and so on.

But that's only going to work if the cache looks like: "h", "hu", "hun", ..., "hunter2"

If just "hunter2" is in the cache, you won't get any signal until you stumble on exactly that password. And that's before getting into the block size granularity of the caches discussed elsewhere in this thread.

That's not to say timing attacks aren't possible. I haven't looked at Claude Code's prompt generation, but there's no intrinsic reason why you couldn't do things like figure out what open source code and research papers your competitors are loading into context.

Sharing caches between orgs would be an incredible misstep.

Re: Prompt caching for cheaper LLM tokens

#64

Post by qeternity »

maxloh wrote: post #50
moebrowne wrote: post #32

Earlier quoted context omitted.

For ChatGPT: > Prompt caches are not shared between organizations. Only members of the same organization can access caches of identical prompts. https://platform.openai.com/docs/guides/prompt-caching#frequ...

I don't find it really viable. There are so many ways to express the same question, and context does matter: the same prompt becomes irrelevant if the previous prompts or LLM responses differ. With the cache limited to the same organization, the chances of it actually being reused would be extremely low.

In a chat setting you hit the cache every time you add a new prompt: all historical question/answer pairs are part of the context and don’t need to be prefilled again.

On the API side imagine you are doing document processing and have a 50k token instruction prompt that you reuse for every document.

It’s extremely viable and used all the time.

Re: Prompt caching for cheaper LLM tokens

#65

Post by jonhohle »

qeternity wrote: post #64
maxloh wrote: post #50

Earlier quoted context omitted.

I don't find it really viable. There are so many ways to express the same question, and context does matter: the same prompt becomes irrelevant if the previous prompts or LLM responses differ. With the cache limited to the same organization, the chances of it actually being reused would be extremely low.

In a chat setting you hit the cache every time you add a new prompt: all historical question/answer pairs are part of the context and don’t need to be prefilled again. On the API side imagine you are doing document processing and have a 50k token instruction prompt that you reuse for every document. It’s extremely viable and used all the time.

I’m shocked that this hasn’t been a thing from the start. That seems like table stakes for automating repetitive tasks.

Re: Prompt caching for cheaper LLM tokens

#66

Post by jgeralnik »

reitzensteinm wrote: post #63
gwern wrote: post #60

Earlier quoted context omitted.

No, the search space is tiny: you can just attack 1 BPE at a time! Stuff like password guessing is almost trivial when you get to do a timing attack on each successive character. So that lets you quickly exfiltrate arbitrary numbers of prompts, especially if you have any idea what you are looking for. (Note that a lot of prompts are already public information, or you can already exfiltrate prompts quite easily from s…

Hill climbing a password would only be possible if intermediate KV cache entries were stored. To hillclimb "hunter2", you're going to try "a", "b", "c", etc, until you notice that "h" comes back faster. Then you try "ha", "hb" and so on. But that's only going to work if the cache looks like: "h", "hu", "hun", ..., "hunter2" If just "hunter2" is in the cache, you won't get any signal until you stumble on exactly that…

Right, you can’t actually guess a letter (byte) at a time but you can guess a token at a time (I believe the vocabulary is 200000 possible tokens in gpt 5) So you could send each of the 200000 possible tokens, see which is cached, and then send 200000 more tokens to find the next cached token Certainly less efficient but well within the realm of a feasible attack

Re: Prompt caching for cheaper LLM tokens

#67

Post by jgeralnik »

IanCal wrote: post #61
gwern wrote: post #60

Earlier quoted context omitted.

No, the search space is tiny: you can just attack 1 BPE at a time! Stuff like password guessing is almost trivial when you get to do a timing attack on each successive character. So that lets you quickly exfiltrate arbitrary numbers of prompts, especially if you have any idea what you are looking for. (Note that a lot of prompts are already public information, or you can already exfiltrate prompts quite easily from s…

Do any providers do this level of granularity? Anthropic require explicit cache markers, for example.

Anthropic requires explicit cache markers but will “look backwards” some amount, so you don’t need to fall on the exact split to get cached tokens

Re: Prompt caching for cheaper LLM tokens

#68

Post by qeternity »

jonhohle wrote: post #65
qeternity wrote: post #64

Earlier quoted context omitted.

In a chat setting you hit the cache every time you add a new prompt: all historical question/answer pairs are part of the context and don’t need to be prefilled again. On the API side imagine you are doing document processing and have a 50k token instruction prompt that you reuse for every document. It’s extremely viable and used all the time.

I’m shocked that this hasn’t been a thing from the start. That seems like table stakes for automating repetitive tasks.

It has been a thing. In a single request, this same cache is reused for each forward pass.

It took a while for companies to start metering it and charging accordingly.

Also companies invested in hierarchical caches that allow longer term and cross cluster caching.

Re: Prompt caching for cheaper LLM tokens

#69

Post by reitzensteinm »

jgeralnik wrote: post #66
reitzensteinm wrote: post #63

Earlier quoted context omitted.

Hill climbing a password would only be possible if intermediate KV cache entries were stored. To hillclimb "hunter2", you're going to try "a", "b", "c", etc, until you notice that "h" comes back faster. Then you try "ha", "hb" and so on. But that's only going to work if the cache looks like: "h", "hu", "hun", ..., "hunter2" If just "hunter2" is in the cache, you won't get any signal until you stumble on exactly that…

Right, you can’t actually guess a letter (byte) at a time but you can guess a token at a time (I believe the vocabulary is 200000 possible tokens in gpt 5) So you could send each of the 200000 possible tokens, see which is cached, and then send 200000 more tokens to find the next cached token Certainly less efficient but well within the realm of a feasible attack

It's a good call out re: tokens vs letters, but I think you might have misunderstood my point - you can't do it a token at a time unless the intermediate KV cache is stored after each token is generated.

This won't be the case in any non toy implementation, as it would be unneccessary and slow.

Re: Prompt caching for cheaper LLM tokens

#70

Post by sroussey »

samwho wrote: post #37
gunalx wrote: post #36

Earlier quoted context omitted.

I habe come across turning on caching means the llm has a faint memory of what was in the cache, even to unrelated queries. If this is the case its fully unreasonable to share the cache, because of possibility of information leakage.

How would information leak, though? There’s no difference in the probability distribution the model outputs when caching vs not caching.

the probability distribution the model outputs is identical under identical conditions.

A local model running alone on your machine will 100% always return the exact same thing and the internal state will be exactly the same and you can checkpoint or cache that to avoid rerunning to that point.

But… conditions can be different, and batching requests tends to affect other items in flight. I believe Thinking Machines had an article about how to make a request deterministic again without performance going to complete crap.

I tend to think of things this way (completely not what happens though): what if you were to cache based on a tensor as the key? To generate a reasonably sized key what is an acceptable loss of precision to retrieve the same cache knowing that there is inherent jitter in the numbers of the tensor?

And then the ever so slight leak of information. But also multiplied since there are internal kv caches for tokens and blah blah blah.

Post reply on HN