Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Caching

docs.anthropic.com

Post reply on HN
Print view View original

51–60 of 75 posts

Re: Prompt Caching

#51

Post by hansonw »

bastawhiz wrote: post #48
GaggiX wrote: post #42

Earlier quoted context omitted.

What is stored is not the tokens, but all keys and values of all attention layers for each token.

As I pointed out, even if it's a gig of data that's still almost an order of magnitude more than the cost of a managed in-memory cache in the cloud. That's wild.

It’s probably more. Pretty conservatively, if the KV embedding dimension for each token is ~10K x 100 attention layers (this is roughly the scale of Llama3.1 405B) that’s already 1M 16-bit floats per token = 2MB. They have likely needed to implement some kind of KV compression (like DeepSeek) to make this even feasible.

Re: Prompt Caching

#52

Post by IanCal »

bastawhiz wrote: post #46
IanCal wrote: post #41

Earlier quoted context omitted.

Is that fast enough? And how much data is being stored? They're not storing the tokens you pass on but the activations after processing them. I'll take a wild stab that the activations for Claude 3.5 aren't anywhere near 4 meg.

You've already frowned on my comparison to S3 but I think it's apt: it's many times more expensive than S3, but (for even a gigabyte of activations) it doesn't even need to be two orders of magnitude faster than standard S3. If you use the Elasticache pricing, which is $0.125/gb per hour, it's still eight times more expensive. So even if a million tokens is a full gigabyte of data, it's still almost an order of magni…

I do not understand your first paragraph. It's more expensive than s3 and it's entirely different. It's cheaper than writing on vellum, and faster!

> If you use the Elasticache pricing, which is $0.125/gb per hour, it's still eight times more expensive. So even if a million tokens is a full gigabyte of data

Is it a gigabyte of data and is it fast enough?

You've guessed 4mb and 1gb. What's the actual data size here? What speed do you need to get it into the GPU ram?

The entire point here is to lower latency and costs so it has to be close and fast.

Guessing at sizes isn't helping anything here.

Re: Prompt Caching

#53

Post by w10-1 »

Setting aside efficiency or accuracy, caching enhances the value of prompt engineering and thus increases the effective value of AI services (how the value is monetized or split is TBD).

Comments suggest that caching the state of the network might also reduce processing.

I wonder if it also permits better A/B-style testing by reducing the effect of cross-domain errors. If the AI service providers made it easy to provide feedback on post-cache responses, the providers could incorporate the quality-enhancement loop accelerating time to product-market fit (at the risk of increasing dependency and reducing ability to switch).

Re: Prompt Caching

#54

Post by jncfhnb »

danielmarkbruce wrote: post #33
jncfhnb wrote: post #20

Earlier quoted context omitted.

I think most architectures do a layer of normalization on the whole text embeddings before calculating attention which makes this infeasible Shouldn’t be a huge deal to adjust imo One of the bigger problems is that closed model providers don’t want to expose the embedding space and let’s users see what they have

I don't think the normalization makes it infeasible. They should be able to make an adjustment (the reverse of the normalization) in one operation. I think they are caching the attention calcs. The hard thing (I think) is what to keep in the cache and where to keep it given you are serving lots of customers and the attention calc can be a large set of numbers pretty quickly.

It’s only “hard” because they don’t want to let customers supply the cache of course

Re: Prompt Caching

#55

Post by danielmarkbruce »

jncfhnb wrote: post #54
danielmarkbruce wrote: post #33

Earlier quoted context omitted.

I don't think the normalization makes it infeasible. They should be able to make an adjustment (the reverse of the normalization) in one operation. I think they are caching the attention calcs. The hard thing (I think) is what to keep in the cache and where to keep it given you are serving lots of customers and the attention calc can be a large set of numbers pretty quickly.

It’s only “hard” because they don’t want to let customers supply the cache of course

I'm not sure that's it. Presumably they want to keep the cache in GPU memory?

Re: Prompt Caching

#56

Post by jncfhnb »

danielmarkbruce wrote: post #55
jncfhnb wrote: post #54

Earlier quoted context omitted.

It’s only “hard” because they don’t want to let customers supply the cache of course

I'm not sure that's it. Presumably they want to keep the cache in GPU memory?

That’s largely it imo. If you get get the embedding representations you could just recreate the model logic and then it’s no longer closed source.

Re: Prompt Caching

#57

Post by danielmarkbruce »

jncfhnb wrote: post #56
danielmarkbruce wrote: post #55

Earlier quoted context omitted.

I'm not sure that's it. Presumably they want to keep the cache in GPU memory?

That’s largely it imo. If you get get the embedding representations you could just recreate the model logic and then it’s no longer closed source.

no, you couldn't. If they just handed you the embedding layer it wouldn't help that much.

Plus, I'm reasonably certain they are caching the attention scores anyway.

Re: Prompt Caching

#59

Post by rfoo »

bastawhiz wrote: post #48
GaggiX wrote: post #42

Earlier quoted context omitted.

What is stored is not the tokens, but all keys and values of all attention layers for each token.

As I pointed out, even if it's a gig of data that's still almost an order of magnitude more than the cost of a managed in-memory cache in the cloud. That's wild.

It's more likely some 400GB of data.

Re: Prompt Caching

#60

Post by karmasimida »

bastawhiz wrote: post #37
verdverm wrote: post #27

FWIW, Gemini / Vertex has this as well and lets you control the TTL. Billing is based on how long you keep the context https://ai.google.dev/gemini-api/docs/caching?lang=python Costs $1 / 1M / 1h

That pricing is ridiculous. A token is essentially a 32 bit integer. Four bytes. A million tokens is 4MB. Imagine paying $1/hr for less than the storage of three floppies. That's two million times more expensive than the storage cost of standard S3 (720 hours×256M tokens (1gb)×$1 vs $0.09). Or 2000 times more expensive than the storage cost of Elasticache serverless. (Yes, I realize it's probably more than 4MB, but i…

You need to cache all the key/values on each layer, otherwise there is no point in caching.
Post reply on HN