High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction
1–10 of 19 posts
Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction
#2I’ve been exploring KV cache optimization for LLM inference.
Most methods (Top-K, sliding window) prune tokens. This works on average, but fails selectively — a few tokens cause large errors when removed.
I tried reframing the problem as approximating the attention function: Attn(Q, K, V)
Prototype: - entropy → identify weak tokens - OLS → reconstruct their contribution - SVD → compress them
Early results show lower error than Top-K at low memory, sometimes even lower memory overall.
This is still a small research prototype, would appreciate feedback or pointers to related work.
Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction
#3Interesting Approach. Curious about the latency tradeoff: OLS + SVD are much heavier than Top-K.Have you benchmarked end-to-end inference latency?
Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction
#4Interesting Approach. Curious about the latency tradeoff: OLS + SVD are much heavier than Top-K.Have you benchmarked end-to-end inference latency?
In this prototype, OLS + SVD isn’t per-token, it runs only when the recycle bin fills (amortized over multiple tokens).
That said, it’s still heavier than Top-K. I haven’t benchmarked end-to-end latency yet; this is mainly exploring the accuracy vs memory tradeoff.
Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction
#5Were there any downsides or difficulties?
It would be sort of surprising if an SVD-based opportunity was missed (since it is such a familiar tool). But, your entropy and least-squares ideas are necessary to set that up, so I guess it makes sense that you’d find some new territory here.
Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction
#6Interesting direction. One question: How does this hold up outside the synthetic transformer on a real downstream task? Reconstruction error is the right measure but its one step removed from the end task. I'm curious whether HAE would show a similar gap on a downstream benchmark.
Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction
#7Interesting Approach. Curious about the latency tradeoff: OLS + SVD are much heavier than Top-K.Have you benchmarked end-to-end inference latency?
From the conclusion:
> The primary trade-off observed is the increased calculation time for OLS and SVD steps. Consequently, the next phase of this work involves implementing these operations within custom Triton kernels to amortize latency. By viewing the cache through the lens of reconstruction fidelity rather than just memory capacity, we can develop more sustainable architectures for long-context inference.
Reading between the lines, the increase in latency was so significant that they didn't want to include it before they had a chance to try and optimize the problem away first.
Still interesting research. Hope they get good results!
Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction
#8I notice the experiments are all run with Gaussian token embeddings and weight matrices, which is a very different scenario than you would get in a real model. It shouldn't be much more difficult to try this with an actual model and data and get a much better sense of how well it compresses.
Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction
#9Interesting direction. One question: How does this hold up outside the synthetic transformer on a real downstream task? Reconstruction error is the right measure but its one step removed from the end task. I'm curious whether HAE would show a similar gap on a downstream benchmark.
[dead]
Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction
#10After a certain amount of context usage, I think I empirically see the stated issues with Top-K compression strategy. It doesn't catastrophically forget but nuances fade as I reach towards the tail end of my context limits.