Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Prompt Caching

docs.anthropic.com

Post reply on HN
Print view View original

21–30 of 75 posts

Re: Prompt Caching

#21

Post by social_quotient »

config_yml wrote: post #13
NBJack wrote: post #2

Sounds kinda useless, TBH. This sounds as if it assumes the exact same context window across requests. If so, given the 5 minute window, unless for example your entire team is operating in the same codebase at the same time, you won't really see any savings beyond simple prompts. Are contexts included in the prompt cache? Are they identified as the same or not? What happens if we approach the 10k token range? 128k? 1…

I have a system prompt of around 100k which includes a couple of internal schema definitions. Having this cached could be super useful to us.

Would love if you could de identify and share a bit more detail on a prompt that big.

Re: Prompt Caching

#22

Post by Scene_Cast2 »

pclmulqdq wrote: post #10
Scene_Cast2 wrote: post #7

Why does prompt caching reduce costs? I'm assuming that the primary cost driver is GPU/TPU FLOPS, as opposed to any network / storage / etc costs. My understanding is that an LLM will take in the stream of text, tokenize it (can be faster with caching, sure, but it's a minor drop in the bucket), then run a transformer on the entire sequence. You can't just cache the output of a transformer on a prefix to reduce workl…

You actually can cache the "output" of a transformer on the prefix by caching what happens in the attention layer for that text string (specifically the "K" and "V" tensors). Since the attention layer is a big part of the compute cost of the transformer, this does cut down FLOPs dramatically.

Oh interesting, didn't know. How does this work past the first transformer in the stack?

Re: Prompt Caching

#23

Post by Sakos »

NBJack wrote: post #2

Sounds kinda useless, TBH. This sounds as if it assumes the exact same context window across requests. If so, given the 5 minute window, unless for example your entire team is operating in the same codebase at the same time, you won't really see any savings beyond simple prompts. Are contexts included in the prompt cache? Are they identified as the same or not? What happens if we approach the 10k token range? 128k? 1…

It says the timeout is refreshed if there's a cache hit within the 5 minutes, and the base cost is 10% of what it would be otherwise. Seems pretty damn useful to me. What seems useless to you exactly?

I'm primarily limited by how much context I need for my queries, and for the majority of the time, the context can often largely be the same across multiple queries over periods of 1-60 minutes. This is the case whether it's a codebase I'm working with or a PDF (or other form of text documentation).

Simple queries are where I expect there to be the least gain for this kind of thing.

Re: Prompt Caching

#24

Post by nprateem »

I just tried Claude the other day. What a breath of fresh air after fighting the dogshit that is OpenAI.

Far less "in the realm of", "in today's fast-moving...", multifaceted, delve or other pretentious wank.

There is still some though so they obviously used the same dataset that's overweight in academic papers. Still, I'm hopeful I can finally get it to write stuff that doesn't sound like AI garbage.

Kind of weird there's no moderation API though. Will they just cut me off if my customers try to write about things they don't like?

Re: Prompt Caching

#25

Post by lonk11 »

Scene_Cast2 wrote: post #22
pclmulqdq wrote: post #10

Earlier quoted context omitted.

You actually can cache the "output" of a transformer on the prefix by caching what happens in the attention layer for that text string (specifically the "K" and "V" tensors). Since the attention layer is a big part of the compute cost of the transformer, this does cut down FLOPs dramatically.

Oh interesting, didn't know. How does this work past the first transformer in the stack?

My understanding is that the attention in all transformer layers is "causal" - that is the output of a transformer layer for token N depends only on tokens from 0 to N.

This means that every attention layer can use previously calculated outputs for the same prompt prefix. So it only needs to calculate from scratch starting from the first unique token in the prompt sequence.

Re: Prompt Caching

#26

Post by Tiberium »

NBJack wrote: post #2

Sounds kinda useless, TBH. This sounds as if it assumes the exact same context window across requests. If so, given the 5 minute window, unless for example your entire team is operating in the same codebase at the same time, you won't really see any savings beyond simple prompts. Are contexts included in the prompt cache? Are they identified as the same or not? What happens if we approach the 10k token range? 128k? 1…

>it assumes the exact same context window across requests That is not true, caching works across multiple requests, that's why it's so good. You can do 5 different concurrent requests and they'll all get cached and cache read if the cache is still warm for them.

Re: Prompt Caching

#28

Post by vellum »

nprateem wrote: post #24

I just tried Claude the other day. What a breath of fresh air after fighting the dogshit that is OpenAI. Far less "in the realm of", "in today's fast-moving...", multifaceted, delve or other pretentious wank. There is still some though so they obviously used the same dataset that's overweight in academic papers. Still, I'm hopeful I can finally get it to write stuff that doesn't sound like AI garbage. Kind of weird t…

You can try AWS Bedrock or Openrouter if that happens. They both have the Claude API.

Re: Prompt Caching

#30

Post by Onavo »

cma wrote: post #19
brylie wrote: post #16

Earlier quoted context omitted.

I’m building an app with OpenAI, using structured outputs. Does OpenAI also support prompt caching?

I'm sure internally they use it for the system prompt at least, probably since launch. And maybe for common initial user queries that exactly match.

They are certainly not passing the savings on to the users.
Post reply on HN