Prompt Caching

Post by burtonator » Sun, Aug 18, 2024, 8:09 PM UTC

Scene_Cast2 wrote: post #7

Why does prompt caching reduce costs? I'm assuming that the primary cost driver is GPU/TPU FLOPS, as opposed to any network / storage / etc costs. My understanding is that an LLM will take in the stream of text, tokenize it (can be faster with caching, sure, but it's a minor drop in the bucket), then run a transformer on the entire sequence. You can't just cache the output of a transformer on a prefix to reduce workl…

Post by minimaxir » Sun, Aug 18, 2024, 8:14 PM UTC

burtonator wrote: post #12

Scene_Cast2 wrote: post #7

Why does prompt caching reduce costs? I'm assuming that the primary cost driver is GPU/TPU FLOPS, as opposed to any network / storage / etc costs. My understanding is that an LLM will take in the stream of text, tokenize it (can be faster with caching, sure, but it's a minor drop in the bucket), then run a transformer on the entire sequence. You can't just cache the output of a transformer on a prefix to reduce workl…

Autoregressive models can't just resume so they have to re-parse the entire prompt again for each execution. By caching them they resume from where it left off from before thereby completely bypassing all that computation. For large contexts this could save a ton of compute! I think this feature and structured outputs are some of the biggest inventions in LLMs this year.

Post by brylie » Sun, Aug 18, 2024, 8:37 PM UTC

minimaxir wrote: post #14

burtonator wrote: post #12

Earlier quoted context omitted.

Autoregressive models can't just resume so they have to re-parse the entire prompt again for each execution. By caching them they resume from where it left off from before thereby completely bypassing all that computation. For large contexts this could save a ton of compute! I think this feature and structured outputs are some of the biggest inventions in LLMs this year.

Prompt caching has been a thing for LLMs since GPT-2 (e.g. transformers's `use_past=True`), it's more of a surprise that it took this long for the main LLM providers to provide a good implementation.

Post by BonoboIO » Sun, Aug 18, 2024, 8:45 PM UTC

config_yml wrote: post #13

NBJack wrote: post #2

Sounds kinda useless, TBH. This sounds as if it assumes the exact same context window across requests. If so, given the 5 minute window, unless for example your entire team is operating in the same codebase at the same time, you won't really see any savings beyond simple prompts. Are contexts included in the prompt cache? Are they identified as the same or not? What happens if we approach the 10k token range? 128k? 1…

I have a system prompt of around 100k which includes a couple of internal schema definitions. Having this cached could be super useful to us.

Post by jncfhnb » Sun, Aug 18, 2024, 9:03 PM UTC

logicchains wrote: post #11

Scene_Cast2 wrote: post #7

Why does prompt caching reduce costs? I'm assuming that the primary cost driver is GPU/TPU FLOPS, as opposed to any network / storage / etc costs. My understanding is that an LLM will take in the stream of text, tokenize it (can be faster with caching, sure, but it's a minor drop in the bucket), then run a transformer on the entire sequence. You can't just cache the output of a transformer on a prefix to reduce workl…

The transformer only looks backwards, so if the first part of the sequence (the prompt) doesn't change, you don't need to rerun it again on that part, just on the part after it that changed. For use cases with large prompts relative to the output size (e.g. lots of examples in the prompt), this can significantly speed up the workload.