31–40 of 75 posts
Earlier quoted context omitted.
I'm sure internally they use it for the system prompt at least, probably since launch. And maybe for common initial user queries that exactly match.
They are certainly not passing the savings on to the users.
Why does prompt caching reduce costs? I'm assuming that the primary cost driver is GPU/TPU FLOPS, as opposed to any network / storage / etc costs. My understanding is that an LLM will take in the stream of text, tokenize it (can be faster with caching, sure, but it's a minor drop in the bucket), then run a transformer on the entire sequence. You can't just cache the output of a transformer on a prefix to reduce workl…
Earlier quoted context omitted.
The transformer only looks backwards, so if the first part of the sequence (the prompt) doesn't change, you don't need to rerun it again on that part, just on the part after it that changed. For use cases with large prompts relative to the output size (e.g. lots of examples in the prompt), this can significantly speed up the workload.
I think most architectures do a layer of normalization on the whole text embeddings before calculating attention which makes this infeasible Shouldn’t be a huge deal to adjust imo One of the bigger problems is that closed model providers don’t want to expose the embedding space and let’s users see what they have
The hard thing (I think) is what to keep in the cache and where to keep it given you are serving lots of customers and the attention calc can be a large set of numbers pretty quickly.
Earlier quoted context omitted.
You actually can cache the "output" of a transformer on the prefix by caching what happens in the attention layer for that text string (specifically the "K" and "V" tensors). Since the attention layer is a big part of the compute cost of the transformer, this does cut down FLOPs dramatically.
Oh interesting, didn't know. How does this work past the first transformer in the stack?
Sounds kinda useless, TBH. This sounds as if it assumes the exact same context window across requests. If so, given the 5 minute window, unless for example your entire team is operating in the same codebase at the same time, you won't really see any savings beyond simple prompts. Are contexts included in the prompt cache? Are they identified as the same or not? What happens if we approach the 10k token range? 128k? 1…
FWIW, Gemini / Vertex has this as well and lets you control the TTL. Billing is based on how long you keep the context https://ai.google.dev/gemini-api/docs/caching?lang=python Costs $1 / 1M / 1h
(Yes, I realize it's probably more than 4MB, but it's still an outrageously high markup. They could do their own caching, not tell you they're doing it, and keep the difference and make even more money)
I just tried Claude the other day. What a breath of fresh air after fighting the dogshit that is OpenAI. Far less "in the realm of", "in today's fast-moving...", multifaceted, delve or other pretentious wank. There is still some though so they obviously used the same dataset that's overweight in academic papers. Still, I'm hopeful I can finally get it to write stuff that doesn't sound like AI garbage. Kind of weird t…
The response you get back will have a refusal, which is pretty standard
FWIW, Gemini / Vertex has this as well and lets you control the TTL. Billing is based on how long you keep the context https://ai.google.dev/gemini-api/docs/caching?lang=python Costs $1 / 1M / 1h
That pricing is ridiculous. A token is essentially a 32 bit integer. Four bytes. A million tokens is 4MB. Imagine paying $1/hr for less than the storage of three floppies. That's two million times more expensive than the storage cost of standard S3 (720 hours×256M tokens (1gb)×$1 vs $0.09). Or 2000 times more expensive than the storage cost of Elasticache serverless. (Yes, I realize it's probably more than 4MB, but i…
Is it a lot for caching in L1 on a chip somewhere? No that'd be wildly cheap.
Is it a lot for "caching" on a tape somewhere? Yes.
So where on this scale does keeping it quick to get to gpu memory lie?
> That's two million times more expensive than the storage cost of standard S3 (
You're not comparing to s3 at all.
Earlier quoted context omitted.
That pricing is ridiculous. A token is essentially a 32 bit integer. Four bytes. A million tokens is 4MB. Imagine paying $1/hr for less than the storage of three floppies. That's two million times more expensive than the storage cost of standard S3 (720 hours×256M tokens (1gb)×$1 vs $0.09). Or 2000 times more expensive than the storage cost of Elasticache serverless. (Yes, I realize it's probably more than 4MB, but i…
Well that really depends where you're caching the data. Is it a lot for caching in L1 on a chip somewhere? No that'd be wildly cheap. Is it a lot for "caching" on a tape somewhere? Yes. So where on this scale does keeping it quick to get to gpu memory lie? > That's two million times more expensive than the storage cost of standard S3 ( You're not comparing to s3 at all.
Google can search the entire Internet in a fraction of a second, they can keep a million tokens within a few dozen milliseconds of a GPU for less than a dollar an hour.