Note that any cache (eg LRU-eviction) is just a specific speculative model for future usage :-)
The cache can be backed by hardware/lookup, or by a cheap computation. The line between functions and data is really blurry.
Speculative KV coding: losslessly compressing KV cache by up to ~4×
11–20 of 50 posts
Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×
#12The problem with this approach is that even recomputing a "draft" of the KV cache is still quadratic in context length. Maybe you can get some constant savings by always recomputing the earliest tokens, but it's not a good tradeoff as context sizes grow.
Sure, but any classical attention mechanism is quadratic in context length.
But text generation is quadratic after the KV cache optimization. If every decode step now has to recompute KV cache including its latest and most expensive tokens (even with a quick, "draft" model) that's even worse.
Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×
#13[dead]
Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×
#14Note that any cache (eg LRU-eviction) is just a specific speculative model for future usage :-) The cache can be backed by hardware/lookup, or by a cheap computation. The line between functions and data is really blurry.
Would you say it is homoiconic, similar to LISP where the syntax of the language is the AST; so, data can become code (Macros) and code can be data (the S-Expression)?
Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×
#15There is no compression taking place here.
Isn't that nitpicking? It's a smaller representation of the data, if you have a certain appetite for decompression time. It could conceivably be worth it. I think it would make a great level 2 cache for older chats.
Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×
#16You can use the original model to compress the kv cache and get ∞x compression, since the prediction is perfect. The cost is time, and I don't see how this could be worth it.
The tradeoff gets better the bigger your primary model, and probably with bigger batch sizes. The KV cache can consume a lot of expensive VRAM, and the VRAM and compute costs of the predictor model become a small fraction of the cost of the primary model For serving a 1T model with 16 concurrent requests this could make a lot of sense. For a 8B model with a single request far less so
This can't be used to save VRAM in practice. To generate a new token with the primary model, you first need to decompress the cache, which involves regenerating the whole sequence from scratch. I.e. generate 1 million tokens with the small model to generate 1 with the large.
Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×
#17How do these results compare with the engram based approach from deepseek?
Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×
#18The problem with this approach is that even recomputing a "draft" of the KV cache is still quadratic in context length. Maybe you can get some constant savings by always recomputing the earliest tokens, but it's not a good tradeoff as context sizes grow.
BTW, I forgot to mention that you can make this work in a way, but only if your model architecture generalizes the context and attention mechanism such that it's no longer a pure sequence. So you could have a large amount of distinct "early" token sequences, with each being self-contained and not depending on any other tokens, e.g. your source code files might be such. Then later parts of the context would of course depend on all of those files as usual. This makes prefill for the earlier context both reusable and cheaply recomputable throughout, at the cost of losing some dependencies that would've been previously accounted for: your model becomes faster and more efficient, but perhaps not quite as smart.
Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×
#19If “speculative” approach works so well in different contexts why not make it first class and use everywhere, possibly recursively?
Speculation is only worth it if you can profit from it. Not every context allows this or has a similar idea of what can be speculated.
It works very well on dense models, imho great alternative to MoE. As verification is cheaper than generation it could be fundamental, first class primitive, maybe even to recurse on it, do live distillation during inference etc.
MoE is more hardcoded, pre determined, speculation is much more dynamic, malleable after training.
This paper actually proposes direction of aligning architecture to aid speculation as future work.
Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×
#20There is no compression taking place here.
Isn't the delta fed to an arithmetic coder?