Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Caching

docs.anthropic.com

Post reply on HN
Print view View original

61–70 of 75 posts

Re: Prompt Caching

#61

Post by karmasimida »

bastawhiz wrote: post #48
GaggiX wrote: post #42

Earlier quoted context omitted.

What is stored is not the tokens, but all keys and values of all attention layers for each token.

As I pointed out, even if it's a gig of data that's still almost an order of magnitude more than the cost of a managed in-memory cache in the cloud. That's wild.

There are some errors in you calculation.

> A token is 32-bit integer.

No, in transformer, token is a vector, for larger models it is probably something like 6k-12k floats, assuming larger model sizes. Assume 8-bit precision, a token is more like 6-12kB, per token.

So assume 100k tokens, you will end up with 554MB for input tokens, ALONE.

Depending on your model architecture, the memory could vary, but from my observation, the runtime memory increase is at least on the same magnitude with the initial amount of memory usage upon loading the model, and this is for a moderate context length (So you are easily looking at caching 10~100GB of data, in a very hot state, and that is going to be very expensive indeed.

Re: Prompt Caching

#62

Post by atgctg »

bastawhiz wrote: post #37
verdverm wrote: post #27

FWIW, Gemini / Vertex has this as well and lets you control the TTL. Billing is based on how long you keep the context https://ai.google.dev/gemini-api/docs/caching?lang=python Costs $1 / 1M / 1h

That pricing is ridiculous. A token is essentially a 32 bit integer. Four bytes. A million tokens is 4MB. Imagine paying $1/hr for less than the storage of three floppies. That's two million times more expensive than the storage cost of standard S3 (720 hours×256M tokens (1gb)×$1 vs $0.09). Or 2000 times more expensive than the storage cost of Elasticache serverless. (Yes, I realize it's probably more than 4MB, but i…

You have to store the KV cache, not the tokens. For Gemma 27B (probably slightly larger than Flash), this would be:

  Size of KV cache = 2 * (num_layers) * (num_kv_heads * dim_head) * seq_length * precision

  8-bit Gemma 27B KV cache = 2 * (46) * (16 * 144) * 1e6 * 1 byte ≈ 200 GB
Note that this doesn't take further optimizations into account that Google might be using.

Formula: https://developer.nvidia.com/blog/mastering-llm-techniques-i...

Gemma 27B config: https://huggingface.co/google/gemma-2-27b/blob/main/config.j...

Re: Prompt Caching

#63

Post by manojlds »

atgctg wrote: post #62
bastawhiz wrote: post #37

Earlier quoted context omitted.

That pricing is ridiculous. A token is essentially a 32 bit integer. Four bytes. A million tokens is 4MB. Imagine paying $1/hr for less than the storage of three floppies. That's two million times more expensive than the storage cost of standard S3 (720 hours×256M tokens (1gb)×$1 vs $0.09). Or 2000 times more expensive than the storage cost of Elasticache serverless. (Yes, I realize it's probably more than 4MB, but i…

You have to store the KV cache, not the tokens. For Gemma 27B (probably slightly larger than Flash), this would be: Size of KV cache = 2 * (num_layers) * (num_kv_heads * dim_head) * seq_length * precision 8-bit Gemma 27B KV cache = 2 * (46) * (16 * 144) * 1e6 * 1 byte ≈ 200 GB Note that this doesn't take further optimizations into account that Google might be using. Formula: https://developer.nvidia.com/blog/masterin…

Is there some easy to understand source / paper about how this caching works?

Re: Prompt Caching

#65

Post by Kiro »

bastawhiz wrote: post #48
GaggiX wrote: post #42

Earlier quoted context omitted.

What is stored is not the tokens, but all keys and values of all attention layers for each token.

As I pointed out, even if it's a gig of data that's still almost an order of magnitude more than the cost of a managed in-memory cache in the cloud. That's wild.

It's not. You're confused and don't understand what you're talking about. Just read all the replies you've already gotten and try to understand why you're wrong instead of doubling down on an incorrect take.

Re: Prompt Caching

#66

Post by kevsim »

politelemon wrote: post #58

Will this be making its way to Bedrock?

How is the Anthropic stuff on Bedrock in general? We're using OpenAI stuff on Azure right now and it's frustrating how slowly stuff gets rolled out in our region(s).

Re: Prompt Caching

#67

Post by danielmarkbruce »

manojlds wrote: post #63
atgctg wrote: post #62

Earlier quoted context omitted.

You have to store the KV cache, not the tokens. For Gemma 27B (probably slightly larger than Flash), this would be: Size of KV cache = 2 * (num_layers) * (num_kv_heads * dim_head) * seq_length * precision 8-bit Gemma 27B KV cache = 2 * (46) * (16 * 144) * 1e6 * 1 byte ≈ 200 GB Note that this doesn't take further optimizations into account that Google might be using. Formula: https://developer.nvidia.com/blog/masterin…

Is there some easy to understand source / paper about how this caching works?

Ask chat gpt to explain how K-V caching works. What they are doing is essentially the same thing, with a few more engineering details.

Re: Prompt Caching

#68

Post by meiraleal »

verdverm wrote: post #27

FWIW, Gemini / Vertex has this as well and lets you control the TTL. Billing is based on how long you keep the context https://ai.google.dev/gemini-api/docs/caching?lang=python Costs $1 / 1M / 1h

So in your calculation all the money spent to create the tech is free?

Re: Prompt Caching

#69

Post by verdverm »

meiraleal wrote: post #68
verdverm wrote: post #27

FWIW, Gemini / Vertex has this as well and lets you control the TTL. Billing is based on how long you keep the context https://ai.google.dev/gemini-api/docs/caching?lang=python Costs $1 / 1M / 1h

So in your calculation all the money spent to create the tech is free?

The "cost" is the published price that Google charges for Gemini caching

Re: Prompt Caching

#70

Post by meiraleal »

verdverm wrote: post #69
meiraleal wrote: post #68

Earlier quoted context omitted.

So in your calculation all the money spent to create the tech is free?

The "cost" is the published price that Google charges for Gemini caching

By this logic the Windows or MS office should have the price slightly higher than an USB sticker
Post reply on HN