Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt caching for cheaper LLM tokens

ngrok.com

Post reply on HN
Print view View original

71–76 of 76 posts

Re: Prompt caching for cheaper LLM tokens

#71

Post by weird-eye-issue »

psadri wrote: post #52
weird-eye-issue wrote: post #40

Earlier quoted context omitted.

They absolutely are segregated With OpenAI at least you can specify the cache key and they even have this in the docs: Use the prompt_cache_key parameter consistently across requests that share common prefixes. Select a granularity that keeps each unique prefix-prompt_cache_key combination below 15 requests per minute to avoid cache overflow.

Does anyone actually compute / use this key feature? Or do you rely on implicit caching? I wish HN had a comment with a poll feature.

It would be important to use for relatively high traffic use cases

Let's say you have a chatbot with hundreds of active users, their requests could get routed to different machines which would mean the implicit caching wouldn't work

If you set the cache key to a user id then it would be more likely each user's chat could get cached on subsequent requests

Re: Prompt caching for cheaper LLM tokens

#72

Post by jgeralnik »

reitzensteinm wrote: post #69
jgeralnik wrote: post #66

Earlier quoted context omitted.

Right, you can’t actually guess a letter (byte) at a time but you can guess a token at a time (I believe the vocabulary is 200000 possible tokens in gpt 5) So you could send each of the 200000 possible tokens, see which is cached, and then send 200000 more tokens to find the next cached token Certainly less efficient but well within the realm of a feasible attack

It's a good call out re: tokens vs letters, but I think you might have misunderstood my point - you can't do it a token at a time unless the intermediate KV cache is stored after each token is generated. This won't be the case in any non toy implementation, as it would be unneccessary and slow.

Ah, fair enough. Anthropic caches at a block level (basically a single message) so for non-trivial messages this is really less of a concern, although I definitely understand why they still scope cache to a single tenant

Re: Prompt caching for cheaper LLM tokens

#73

Post by ambicapter »

weird-eye-issue wrote: post #40
samwho wrote: post #30

Earlier quoted context omitted.

I was wondering about this when I was reading around the topic. I can’t personally think of a reason you would need to segregate, though it wouldn’t surprise me if they do for some sort of compliance reasons. I’m not sure though, would love to hear something first-party.

They absolutely are segregated With OpenAI at least you can specify the cache key and they even have this in the docs: Use the prompt_cache_key parameter consistently across requests that share common prefixes. Select a granularity that keeps each unique prefix-prompt_cache_key combination below 15 requests per minute to avoid cache overflow.

> Select a granularity that keeps each unique prefix-prompt_cache_key combination below 15 requests per minute to avoid cache overflow.

Why below a certain number? Usually in caches a high number of requests keeps the cached bit from expiring or being replaced, no?

Re: Prompt caching for cheaper LLM tokens

#74

Post by NooneAtAll3 »

samwho wrote: post #21
NooneAtAll3 wrote: post #20

Blog starts loading and then gives "Something Went Wrong. D is not a function" error displayed

Could you tell me what browser/OS/device you’re using? A few people have said this and I haven’t been able to reproduce it.

librewolf, fork of firefox, latest version

f12 menu lists this:

Loading failed for the with source “https://global.ketchcdn.com/web/v2/config/ngrok/ngrok_ketch_...”. prompt-caching:1:356 Response { status: 404, type: "default", url: "", redirected: false, ok: false, statusText: "Not Found", headers: Headers(1), body: ReadableStream, bodyUsed: false }

React Router caught the following error during render entry.client-BTJ7ChVH.js:8:64676 Response { status: 404, type: "default", url: "", redirected: false, ok: false, statusText: "Not Found", headers: Headers(1), body: ReadableStream, bodyUsed: false }

Uncaught Error: Minified React error #520; visit https://react.dev/errors/520 for the full message or use the non-minified dev environment for full errors and additional helpful warnings. chunk-G3INQAYP-D7BZozYw.js:4:2490 Rm https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... mu https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... Lm https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... t1 https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... A1 https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... Ba https://frontend-blog-ngrok.vercel.app/assets/entry.client-B... Caused by: Response { … }

Re: Prompt caching for cheaper LLM tokens

#75

Post by weird-eye-issue »

ambicapter wrote: post #73
weird-eye-issue wrote: post #40

Earlier quoted context omitted.

They absolutely are segregated With OpenAI at least you can specify the cache key and they even have this in the docs: Use the prompt_cache_key parameter consistently across requests that share common prefixes. Select a granularity that keeps each unique prefix-prompt_cache_key combination below 15 requests per minute to avoid cache overflow.

> Select a granularity that keeps each unique prefix-prompt_cache_key combination below 15 requests per minute to avoid cache overflow. Why below a certain number? Usually in caches a high number of requests keeps the cached bit from expiring or being replaced, no?

It needs to go to the same machine and machines can only handle so many requests
Post reply on HN