21–30 of 30 posts
Does anyone have a good recommendation for explaining or as a primer on KV cache?
For something gentler, 3Blue1Brown: https://www.youtube.com/watch?v=eMlx5fFNoYc (this is part of a series)
Prefix caching is already widely deployed by all providers, right? llama.cpp does it. vLLM does it. I'm sure everyone hosting LLMs for a living does it. This paper seems to focus entirely on prefixes (i.e. the prefilled content is rooted at 0). This is... nothing. The referenced CacheBlend paper ( https://arxiv.org/pdf/2405.16444 ) which tries to stitch together multiple independent prefills looks more interesting an…
The KV cache is order dependent and dependent on the context of tokens which exist before the KV cache. There are some transformation approaches to re-use the kv cache across inferences, but none are in wide use due to accuracy concerns following the transformation.
Absolute slop paper. Replace document with text and you'll get it. "People are asking the same questions and an answer is generated every time, what if we could like cache the questions and their answers..." Sounds like someone was using chatgpt to understand how chatgpt works and then asked it to generate a paper based on his proposal to improve it.
Prefix caching is already widely deployed by all providers, right? llama.cpp does it. vLLM does it. I'm sure everyone hosting LLMs for a living does it. This paper seems to focus entirely on prefixes (i.e. the prefilled content is rooted at 0). This is... nothing. The referenced CacheBlend paper ( https://arxiv.org/pdf/2405.16444 ) which tries to stitch together multiple independent prefills looks more interesting an…
Yeah, I'm really not sure what the point of this paper is. Every non-toy environment does prefix caching.
The paper's approach should work well if (a) you can calculate KV(A || B) as a function of KV(A) and KV(B) independently, (b) you can identify which documents A1, A2, A3, ... are used commonly enough to be worth caching, and (c) it is cheaper to buy and sell KV(A) on a market than to compute KV(A) when it is needed. Given the size of KV(A) I am not sure that (c) will become true even if people solve the open research problem represented by (a) and accept the state-of-the-art trade-offs known for (b).
The KV cache is order dependent and dependent on the context of tokens which exist before the KV cache. There are some transformation approaches to re-use the kv cache across inferences, but none are in wide use due to accuracy concerns following the transformation.
My understanding was that what the KV cache stores is nothing else than the "activations" of the W_k and W_v matrices of an attention module for a given input sequence.
So I don't quite understand how this is supposed to work:
> Let a publisher precompute a document's KV cache, and let every other agent buy the right to load it and skip prefill.
Should a publisher precompute the cache for every popular model that is out there?
> Then the part that matters: where the KV lives When your abstract was clearly generated by an LLM and not curated to at least make it sound human, it does not make me want to read your paper.
especially because this is the most painfully glaring flaw in their plan. Their solution is for an inference provider to... store the KV cache (which they can compute!) on-premise, on their own disks, but pay some third party for it?
KV caching is a super interesting engineering space, especially when you’re talking about local models where compute and memory bandwidth are highly constrained and you’re trying to trim fractions of a second everywhere you can by flipping between different ICL prefixes. But selling caches for specific documents just makes no sense at all.
Earlier quoted context omitted.
Yeah, I'm really not sure what the point of this paper is. Every non-toy environment does prefix caching.
Yes, but presumably the authors are suggesting broader application than just caching a system prompt. The paper's approach should work well if (a) you can calculate KV(A || B) as a function of KV(A) and KV(B) independently, (b) you can identify which documents A1, A2, A3, ... are used commonly enough to be worth caching, and (c) it is cheaper to buy and sell KV(A) on a market than to compute KV(A) when it is needed.…
The authors of the OP paper "Can I Buy Your KV Cache?" explicitly disregard anything involving KV not rooted at 0:
>> We deliberately study the simplest, safe form: a document treated as a shared prefix, with continuations appended after it
So no, I really think it's just prefix caching. That's actually far from the weirdest thing about that paper: they go on to "prove" that decoding from cached prefill gets the same result as prefilling and decoding on the same content, which... yes. That is how computation works.
Also, the thing they describe already exists: you pay your provider for their cache implementation as part of your token ingress costs. What is that if not paying for cached KV?
Earlier quoted context omitted.
Any keyword or paper I can search for?
AsyncResoning[1] does a trick of that sort to give agents concurrent cache views. You basically have two agents look at the same cache under different views. Say agent_0 gets [a_1, a_0] and agent_1 gets [a_0, a_1]. They also write to this cache concurrently while decoding. To solve positional embedding inconsistencies they rotate the query projections for each block (a_0 and a_1) separately. The computations you get…
Does anyone have a good recommendation for explaining or as a primer on KV cache?
The KV cache is order dependent and dependent on the context of tokens which exist before the KV cache. There are some transformation approaches to re-use the kv cache across inferences, but none are in wide use due to accuracy concerns following the transformation.
Isn't it also, most fundamentally, dependent on the model weights? My understanding was that what the KV cache stores is nothing else than the "activations" of the W_k and W_v matrices of an attention module for a given input sequence. So I don't quite understand how this is supposed to work: > Let a publisher precompute a document's KV cache, and let every other agent buy the right to load it and skip prefill. Shoul…