Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Caching

docs.anthropic.com

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Prompt Caching

#41

Post by IanCal »

bastawhiz wrote: post #40
IanCal wrote: post #39

Earlier quoted context omitted.

Well that really depends where you're caching the data. Is it a lot for caching in L1 on a chip somewhere? No that'd be wildly cheap. Is it a lot for "caching" on a tape somewhere? Yes. So where on this scale does keeping it quick to get to gpu memory lie? > That's two million times more expensive than the storage cost of standard S3 ( You're not comparing to s3 at all.

"RAM near a GPU" is ~the same cost as "RAM near literally any other piece of hardware". Even if it has to traverse the network, that's a fairly low, fairly fixed (in, say, the same rack) cost. Hell, it's probably even fast enough to use an NVME disk. Google can search the entire Internet in a fraction of a second, they can keep a million tokens within a few dozen milliseconds of a GPU for less than a dollar an hour.

Is that fast enough? And how much data is being stored? They're not storing the tokens you pass on but the activations after processing them. I'll take a wild stab that the activations for Claude 3.5 aren't anywhere near 4 meg.

Re: Prompt Caching

#42

Post by GaggiX »

bastawhiz wrote: post #37
verdverm wrote: post #27

FWIW, Gemini / Vertex has this as well and lets you control the TTL. Billing is based on how long you keep the context https://ai.google.dev/gemini-api/docs/caching?lang=python Costs $1 / 1M / 1h

That pricing is ridiculous. A token is essentially a 32 bit integer. Four bytes. A million tokens is 4MB. Imagine paying $1/hr for less than the storage of three floppies. That's two million times more expensive than the storage cost of standard S3 (720 hours×256M tokens (1gb)×$1 vs $0.09). Or 2000 times more expensive than the storage cost of Elasticache serverless. (Yes, I realize it's probably more than 4MB, but i…

What is stored is not the tokens, but all keys and values of all attention layers for each token.

Re: Prompt Caching

#43

Post by sdrg822 »

GaggiX wrote: post #42
bastawhiz wrote: post #37

Earlier quoted context omitted.

That pricing is ridiculous. A token is essentially a 32 bit integer. Four bytes. A million tokens is 4MB. Imagine paying $1/hr for less than the storage of three floppies. That's two million times more expensive than the storage cost of standard S3 (720 hours×256M tokens (1gb)×$1 vs $0.09). Or 2000 times more expensive than the storage cost of Elasticache serverless. (Yes, I realize it's probably more than 4MB, but i…

What is stored is not the tokens, but all keys and values of all attention layers for each token.

+1 it wouldn’t be terribly useful if it were only caching the tokenizer output.

Re: Prompt Caching

#44

Post by HeatrayEnjoyer »

Onavo wrote: post #30
cma wrote: post #19

Earlier quoted context omitted.

I'm sure internally they use it for the system prompt at least, probably since launch. And maybe for common initial user queries that exactly match.

They are certainly not passing the savings on to the users.

That $100 billion data center has to get paid for somehow.

Re: Prompt Caching

#46

Post by bastawhiz »

IanCal wrote: post #41
bastawhiz wrote: post #40

Earlier quoted context omitted.

"RAM near a GPU" is ~the same cost as "RAM near literally any other piece of hardware". Even if it has to traverse the network, that's a fairly low, fairly fixed (in, say, the same rack) cost. Hell, it's probably even fast enough to use an NVME disk. Google can search the entire Internet in a fraction of a second, they can keep a million tokens within a few dozen milliseconds of a GPU for less than a dollar an hour.

Is that fast enough? And how much data is being stored? They're not storing the tokens you pass on but the activations after processing them. I'll take a wild stab that the activations for Claude 3.5 aren't anywhere near 4 meg.

You've already frowned on my comparison to S3 but I think it's apt: it's many times more expensive than S3, but (for even a gigabyte of activations) it doesn't even need to be two orders of magnitude faster than standard S3.

If you use the Elasticache pricing, which is $0.125/gb per hour, it's still eight times more expensive. So even if a million tokens is a full gigabyte of data, it's still almost an order of magnitude more expensive than an in-memory cache adjacent to the inference boxes.

When your managed cache costs right times as much as a general purpose managed cache _in the cloud_, you've jumped the shark on pricing.

Re: Prompt Caching

#47

Post by simonw »

NBJack wrote: post #2

Sounds kinda useless, TBH. This sounds as if it assumes the exact same context window across requests. If so, given the 5 minute window, unless for example your entire team is operating in the same codebase at the same time, you won't really see any savings beyond simple prompts. Are contexts included in the prompt cache? Are they identified as the same or not? What happens if we approach the 10k token range? 128k? 1…

It’s useful to consider the impact this has on conversations.

In an ongoing conversation with a model you end up re-submitting the full test of the previous conversation - both prompts and responses - at every step. This means the cost per prompt in that conversation increases each time.

Claude prompt caching can start saving you money even with just a single user having a conversation, provided each of their replies is within five minutes of the previous reply.

Re: Prompt Caching

#48

Post by bastawhiz »

GaggiX wrote: post #42
bastawhiz wrote: post #37

Earlier quoted context omitted.

That pricing is ridiculous. A token is essentially a 32 bit integer. Four bytes. A million tokens is 4MB. Imagine paying $1/hr for less than the storage of three floppies. That's two million times more expensive than the storage cost of standard S3 (720 hours×256M tokens (1gb)×$1 vs $0.09). Or 2000 times more expensive than the storage cost of Elasticache serverless. (Yes, I realize it's probably more than 4MB, but i…

What is stored is not the tokens, but all keys and values of all attention layers for each token.

As I pointed out, even if it's a gig of data that's still almost an order of magnitude more than the cost of a managed in-memory cache in the cloud. That's wild.

Re: Prompt Caching

#49

Post by nextworddev »

WiSaGaN wrote: post #45

This feature was first introduced by deepseek. And deepseek will just do it automatically for you. https://platform.deepseek.com/api-docs/news/news0802/

Yep. All the LLM providers fast follow aka copy each other within months, though

Re: Prompt Caching

#50

Post by GaggiX »

bastawhiz wrote: post #48
GaggiX wrote: post #42

Earlier quoted context omitted.

What is stored is not the tokens, but all keys and values of all attention layers for each token.

As I pointed out, even if it's a gig of data that's still almost an order of magnitude more than the cost of a managed in-memory cache in the cloud. That's wild.

It's probably much more than a single GB of data for a million tokens.
Post reply on HN