Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt caching for cheaper LLM tokens

ngrok.com

Post reply on HN
Print view View original

31–40 of 76 posts

Re: Prompt caching for cheaper LLM tokens

#31

Post by moebrowne »

samwho wrote: post #27
willvarfar wrote: post #25

A really clear explanation! So if I were running a provider I would be caching popular prefixes for questions across all users. There must be so many questions that start 'what is' or 'who was' etc? Also, can subsequences in the prompt be cached and reused? Or is it only prefixes? I mean, can you cache popular phrases that might appear in the middle of the prompt and reuse that somehow rather than needing to iterate…

With KV caching as it’s described there it has to be a prefix match. OpenAI state in their docs they don’t cache anything below 1024 tokens long, and I’m sure I read somewhere that they only cache in 1024 token blocks (so 1024, 2048, 3072, etc) but I can’t find it now. There’s been some research into how to cache chunks in the middle, but I don’t think any of the providers are doing it yet because it needs the prompt…

https://platform.openai.com/docs/guides/prompt-caching#requi...

> Caching is available for prompts containing 1024 tokens or more.

No mention of caching being in blocks of 1024 tokens thereafter.

Re: Prompt caching for cheaper LLM tokens

#32

Post by moebrowne »

Havoc wrote: post #29

Does anyone know whether the cache is segregated by user/API key for the big providers? Was looking at modifying outgoing requests via proxy and wondering whether that's harming caching. Common coding tools presumably have a shared prompt across all their installs so universal cache would save a lot

For ChatGPT:

> Prompt caches are not shared between organizations. Only members of the same organization can access caches of identical prompts.

https://platform.openai.com/docs/guides/prompt-caching#frequ...

Re: Prompt caching for cheaper LLM tokens

#33

Post by samwho »

samwho wrote: post #30
Havoc wrote: post #29

Does anyone know whether the cache is segregated by user/API key for the big providers? Was looking at modifying outgoing requests via proxy and wondering whether that's harming caching. Common coding tools presumably have a shared prompt across all their installs so universal cache would save a lot

I was wondering about this when I was reading around the topic. I can’t personally think of a reason you would need to segregate, though it wouldn’t surprise me if they do for some sort of compliance reasons. I’m not sure though, would love to hear something first-party.

The only thing that comes to mind is some kind of timing attack. Send loads of requests specific to a company you’re trying to spy on and if it comes back cached you know someone has sent that prompt recently. Expensive attack, though, with a large search space.

Re: Prompt caching for cheaper LLM tokens

#34

Post by wesammikhail »

stingraycharles wrote: post #19
wesammikhail wrote: post #8

Earlier quoted context omitted.

Amazing article. I was under the misapprehension that temp and other output parameters actually do affect caching. Turns out I was wrong and this explains why beautifully. Great work. Learned a lot!

I had a “somebody is wrong on the internet!!” discussion about exactly this a few weeks ago, and they proclaimed to be a professor in AI. Where do people get the idea from that temperature affects caching in any way? Temperature is about next token prediction / output, not input.

Because in my mind, as a person not working directly on this kind of stuff, I figured that caching was done similar to any resource caching in a webserver environment.

It´s a semantics issue where the word caching is overloaded depending on context. For people that are not familiar with the inner workings of llm models, this can cause understandable confusion.

Re: Prompt caching for cheaper LLM tokens

#35

Post by wesammikhail »

samwho wrote: post #16
wesammikhail wrote: post #8

Earlier quoted context omitted.

Amazing article. I was under the misapprehension that temp and other output parameters actually do affect caching. Turns out I was wrong and this explains why beautifully. Great work. Learned a lot!

Yay, glad I could help! The sampling process is so interesting on its own that I really want to do a piece on it as well.

Looking forward to it!

Re: Prompt caching for cheaper LLM tokens

#36

Post by gunalx »

samwho wrote: post #33
samwho wrote: post #30

Earlier quoted context omitted.

I was wondering about this when I was reading around the topic. I can’t personally think of a reason you would need to segregate, though it wouldn’t surprise me if they do for some sort of compliance reasons. I’m not sure though, would love to hear something first-party.

The only thing that comes to mind is some kind of timing attack. Send loads of requests specific to a company you’re trying to spy on and if it comes back cached you know someone has sent that prompt recently. Expensive attack, though, with a large search space.

I habe come across turning on caching means the llm has a faint memory of what was in the cache, even to unrelated queries. If this is the case its fully unreasonable to share the cache, because of possibility of information leakage.

Re: Prompt caching for cheaper LLM tokens

#37

Post by samwho »

gunalx wrote: post #36
samwho wrote: post #33

Earlier quoted context omitted.

The only thing that comes to mind is some kind of timing attack. Send loads of requests specific to a company you’re trying to spy on and if it comes back cached you know someone has sent that prompt recently. Expensive attack, though, with a large search space.

I habe come across turning on caching means the llm has a faint memory of what was in the cache, even to unrelated queries. If this is the case its fully unreasonable to share the cache, because of possibility of information leakage.

How would information leak, though? There’s no difference in the probability distribution the model outputs when caching vs not caching.

Re: Prompt caching for cheaper LLM tokens

#40

Post by weird-eye-issue »

samwho wrote: post #30
Havoc wrote: post #29

Does anyone know whether the cache is segregated by user/API key for the big providers? Was looking at modifying outgoing requests via proxy and wondering whether that's harming caching. Common coding tools presumably have a shared prompt across all their installs so universal cache would save a lot

I was wondering about this when I was reading around the topic. I can’t personally think of a reason you would need to segregate, though it wouldn’t surprise me if they do for some sort of compliance reasons. I’m not sure though, would love to hear something first-party.

They absolutely are segregated

With OpenAI at least you can specify the cache key and they even have this in the docs:

Use the prompt_cache_key parameter consistently across requests that share common prefixes. Select a granularity that keeps each unique prefix-prompt_cache_key combination below 15 requests per minute to avoid cache overflow.

Post reply on HN