Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction

jchandra.com

Post reply on HN
Print view View original

11–19 of 19 posts

Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction

#11

Post by jchandra »

bee_rider wrote: post #5

Were there any downsides or difficulties? It would be sort of surprising if an SVD-based opportunity was missed (since it is such a familiar tool). But, your entropy and least-squares ideas are necessary to set that up, so I guess it makes sense that you’d find some new territory here.

That’s a great point and yeah, I’d agree SVD itself isn’t new at all.

On downsides: definitely a few. The biggest one is latency - SVD is fairly heavy, so even though it’s amortized (runs periodically, not per token), it still adds noticeable overhead. It’s also more complex than simple pruning, and I haven’t validated how well this holds on real downstream tasks yet.

This is very much a research prototype right now more about exploring a different tradeoff space than something ready for production.

Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction

#12

Post by jchandra »

aesthesia wrote: post #8

I notice the experiments are all run with Gaussian token embeddings and weight matrices, which is a very different scenario than you would get in a real model. It shouldn't be much more difficult to try this with an actual model and data and get a much better sense of how well it compresses.

I completely agree.Right now this is all on a synthetic setup to isolate the behavior and understand the reconstruction vs memory tradeoff. Real models will definitely behave differently.

I’ve started trying this out with actual models, but currently running things CPU-bound, so it’s pretty slow. Would ideally want to try this properly on GPU, but that gets expensive quickly

So yeah, still very much a research prototype — but validating this on real models/data is definitely the next step.

Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction

#14

Post by jchandra »

scythmic_waves wrote: post #7
vivahir215 wrote: post #3

Interesting Approach. Curious about the latency tradeoff: OLS + SVD are much heavier than Top-K.Have you benchmarked end-to-end inference latency?

From the conclusion: > The primary trade-off observed is the increased calculation time for OLS and SVD steps. Consequently, the next phase of this work involves implementing these operations within custom Triton kernels to amortize latency. By viewing the cache through the lens of reconstruction fidelity rather than just memory capacity, we can develop more sustainable architectures for long-context inference. Readi…

Haha, that’s a very fair reading :)

Yeah, the latency hit is definitely real. That said, most of what I’ve run so far is CPU-bound, which likely exaggerates it quite a bit so I didn’t want to draw strong conclusions from that.

Would need proper GPU implementations to really understand where it lands.

Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction

#16

Post by jchandra »

thw20 wrote: post #15

Good work! This is very interesting. Here's a related work that construct low-rank approximation for attention: https://arxiv.org/abs/2505.12942 . Maybe the idea of Query calibration matrix Rxx is of interest to the author!

Thanks, really appreciate the pointer. Will dig into it.

Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction

#17

Post by jchandra »

jbellis wrote: post #13

Isn't the "KV Compression Strategies (FAIR)" chart showing that the fancy complex algorithm only barely beats simple topk? The commentary says that topk "degrades rapidly at low ratios" but the same can be seen for HAE (Entropy + OLS).

Fair point, the gap isn’t huge in that plot, and both degrade at low ratios. The difference is more in how they degrade: TopK can have sharper, localized failures, while HAE tends to be a bit more smooth. That doesn’t always show up strongly in average MSE.

That said, the gains are modest right now, this is still a research prototype exploring the tradeoff, and there’s clearly more work to be done.

Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction

#18

Post by jchandra »

rishabhaiover wrote: post #10

After a certain amount of context usage, I think I empirically see the stated issues with Top-K compression strategy. It doesn't catastrophically forget but nuances fade as I reach towards the tail end of my context limits.

Yeah, that’s consistent. topK keeps the obvious tokens, but subtle context gets eroded over time rather than dropped all at once.

Re: High-Fidelity KV Cache Summarization Using Entropy and Low-Rank Reconstruction

#19

Post by bee_rider »

jbellis wrote: post #13

Isn't the "KV Compression Strategies (FAIR)" chart showing that the fancy complex algorithm only barely beats simple topk? The commentary says that topk "degrades rapidly at low ratios" but the same can be seen for HAE (Entropy + OLS).

Is it really that fancy and complex, though? The “entropy recycling bin” seems fancy to me, but the other stuff is least squares and an SVD, these are solid workhorse numerical routines.
Post reply on HN