Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt caching for cheaper LLM tokens

ngrok.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: Prompt caching for cheaper LLM tokens

#41

Post by weird-eye-issue »

gunalx wrote: post #36
samwho wrote: post #33

Earlier quoted context omitted.

The only thing that comes to mind is some kind of timing attack. Send loads of requests specific to a company you’re trying to spy on and if it comes back cached you know someone has sent that prompt recently. Expensive attack, though, with a large search space.

I habe come across turning on caching means the llm has a faint memory of what was in the cache, even to unrelated queries. If this is the case its fully unreasonable to share the cache, because of possibility of information leakage.

This is absolutely 100% incorrect.

Re: Prompt caching for cheaper LLM tokens

#42

Post by andruby »

holbrad wrote: post #39

I gave the table of inputs and outputs to both Gemini 3.0 flash and GPT 5.2 instant and they were stumped. https://t3.chat/share/j2tnfwwful https://t3.chat/share/k1xhgisrw1

What is the function supposed to be? It’s not Celsius to Farenheit. (2C=35F, 206C=406F, …)

Re: Prompt caching for cheaper LLM tokens

#43

Post by samwho »

holbrad wrote: post #39

I gave the table of inputs and outputs to both Gemini 3.0 flash and GPT 5.2 instant and they were stumped. https://t3.chat/share/j2tnfwwful https://t3.chat/share/k1xhgisrw1

When I was writing this, GPT 5.1 was the latest and it got it right away. It’s the sequence of prime numbers fwiw :)

Re: Prompt caching for cheaper LLM tokens

#46

Post by yomismoaqui »

samwho wrote: post #17
est wrote: post #5

This is a surprising good read of how LLM works in general.

It’s funny, I didn’t set out for that to be the case. When I pitched the idea internally, I wanted to scratch my own itch (what on earth is a cached token?) and produce a good post. But then I realised I had to go deeper and deeper to get to my answer and accidentally made a very long explainer.

Thanks for the post, it's near perfect in focus, detail and how it's written.

EDIT: You have some minor typos in the post (psuedocode)

Re: Prompt caching for cheaper LLM tokens

#47

Post by bkor »

samwho wrote: post #18
Youden wrote: post #4

Link seems to be broken: content briefly loads then is replaced with "Something Went Wrong" then "D is not a function". Stays broken with adblock disabled.

Another person had this problem as well and we couldn’t figure out what causes it. We suspect something to do with WebGL support. What browser/device are you using? Does it still break if you disable all extensions? I’d love to fix this.

It gives "D is not a function". This on Firefox 146. Various extensions including Ublock Origin but that doesn't seem to cause it. Also doesn't work in a private window.

Re: Prompt caching for cheaper LLM tokens

#48

Post by walterbell »

tomhow wrote: post #24

[under-the-rug stub] [see https://news.ycombinator.com/item?id=45988611 for explanation]

Excellent HN-esque innovation in moderation: immediate improvement in S/N ratio, unobtrusive UX, gentle feedback to humans, semantic signal to machines.

How was the term "rug" chosen, e.g. in the historical context of newspaper folds?

Re: Prompt caching for cheaper LLM tokens

#49

Post by dustfinger »

samwho wrote: post #30
Havoc wrote: post #29

Does anyone know whether the cache is segregated by user/API key for the big providers? Was looking at modifying outgoing requests via proxy and wondering whether that's harming caching. Common coding tools presumably have a shared prompt across all their installs so universal cache would save a lot

I was wondering about this when I was reading around the topic. I can’t personally think of a reason you would need to segregate, though it wouldn’t surprise me if they do for some sort of compliance reasons. I’m not sure though, would love to hear something first-party.

I wonder if there is valuable information that can be learned by studying a companies prompts? There may be reasons why some companies want their prompts private.

Re: Prompt caching for cheaper LLM tokens

#50

Post by maxloh »

moebrowne wrote: post #32
Havoc wrote: post #29

Does anyone know whether the cache is segregated by user/API key for the big providers? Was looking at modifying outgoing requests via proxy and wondering whether that's harming caching. Common coding tools presumably have a shared prompt across all their installs so universal cache would save a lot

For ChatGPT: > Prompt caches are not shared between organizations. Only members of the same organization can access caches of identical prompts. https://platform.openai.com/docs/guides/prompt-caching#frequ...

I don't find it really viable. There are so many ways to express the same question, and context does matter: the same prompt becomes irrelevant if the previous prompts or LLM responses differ.

With the cache limited to the same organization, the chances of it actually being reused would be extremely low.

Post reply on HN