Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speculative KV coding: losslessly compressing KV cache by up to ~4×

fergusfinn.com

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#31

Post by lostmsu »

zozbot234 wrote: post #30
xlayn wrote: post #27

Earlier quoted context omitted.

I created a patch for llama.cpp to store on disk instead of deleting the kv cache as well as the checkpoints... there is this bug on llama.cpp if you have more than one instance going on of chats... and that causes the kv cache to be lost between changes of chat... And I can tell you, using Qwen3.627B after one day of use you can have 120-200Gb of chats on disk. And yes it's way way faster, even if you get it from a…

Qwen 27B maxes out at a 16GB context. A nice thing about DeepSeek V4, especially Flash, is that its context size stays tiny even at 1M tokens! Which in turn opens up wide batching on common consumer platforms.

DeepSeek V4 Flash is 160GB while Qwen 27B is about 27GB. You can't even run DS Flash on consumer platforms, let alone batch it.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#32

Post by nicman23 »

oceanplexian wrote: post #23

A lot of this is over my head but why would you do compression when GPU time is the most expensive thing in the world right now? KV can be trivially stored on ram or even a spinning disk and retrieved on the order of milliseconds. See LM cache for vLLM for example. In fact it’s so easy it kinda shocks me when Claude Code will sit and recompute my entire KV on a new session after a couple of hours, I guess Anthropic i…

edge

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#33

Post by zozbot234 »

lostmsu wrote: post #31
zozbot234 wrote: post #30

Earlier quoted context omitted.

Qwen 27B maxes out at a 16GB context. A nice thing about DeepSeek V4, especially Flash, is that its context size stays tiny even at 1M tokens! Which in turn opens up wide batching on common consumer platforms.

DeepSeek V4 Flash is 160GB while Qwen 27B is about 27GB. You can't even run DS Flash on consumer platforms, let alone batch it.

These are the sizes of model weights, not the KV cache. The former are a sparse (for MoE models) read workload that can be streamed from SSD.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#34

Post by somnial »

zozbot234 wrote: post #6

The problem with this approach is that even recomputing a "draft" of the KV cache is still quadratic in context length. Maybe you can get some constant savings by always recomputing the earliest tokens, but it's not a good tradeoff as context sizes grow.

true, but no reason the predictor model couldn't use linear attention (i.e. mamba, GDN etc) to predict KV caches

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#35

Post by lostmsu »

zozbot234 wrote: post #33
lostmsu wrote: post #31

Earlier quoted context omitted.

DeepSeek V4 Flash is 160GB while Qwen 27B is about 27GB. You can't even run DS Flash on consumer platforms, let alone batch it.

These are the sizes of model weights, not the KV cache. The former are a sparse (for MoE models) read workload that can be streamed from SSD.

You can't batch MoE

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#36

Post by zozbot234 »

lostmsu wrote: post #35
zozbot234 wrote: post #33

Earlier quoted context omitted.

These are the sizes of model weights, not the KV cache. The former are a sparse (for MoE models) read workload that can be streamed from SSD.

You can't batch MoE

You need wider batches to get effective reuse of experts in any given layer, but you absolutely can. DeepSeek V4 has tiny KV caches that make this quite feasible. When targeting consumer platforms that only have a limited amount of compute headroom to begin with, the approach is quite reasonable.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#37

Post by lostmsu »

zozbot234 wrote: post #36
lostmsu wrote: post #35

Earlier quoted context omitted.

You can't batch MoE

You need wider batches to get effective reuse of experts in any given layer, but you absolutely can. DeepSeek V4 has tiny KV caches that make this quite feasible. When targeting consumer platforms that only have a limited amount of compute headroom to begin with, the approach is quite reasonable.

Sounds like you're talking out of your butt instead of doing the math.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#38

Post by zozbot234 »

lostmsu wrote: post #37
zozbot234 wrote: post #36

Earlier quoted context omitted.

You need wider batches to get effective reuse of experts in any given layer, but you absolutely can. DeepSeek V4 has tiny KV caches that make this quite feasible. When targeting consumer platforms that only have a limited amount of compute headroom to begin with, the approach is quite reasonable.

Sounds like you're talking out of your butt instead of doing the math.

What do you mean by doing the math? If you repeatedly sample n_active experts out of n_total, why wouldn't you expect to get some meaningful probability of reuse/overlap once your batch grows past size 5 or so (for the sparsest MoE models in common use)? And you only need enough reuse to fill the compute headroom which is quite small on consumer platforms (we won't have huge TOPS numbers for the typical integrated GPU in Strix Halo or even the upcoming RTX Spark). Plus if you're a single user running multiple streams in parallel the choice of experts will be highly biased leading to more reuse.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#39

Post by lostmsu »

zozbot234 wrote: post #38
lostmsu wrote: post #37

Earlier quoted context omitted.

Sounds like you're talking out of your butt instead of doing the math.

What do you mean by doing the math? If you repeatedly sample n_active experts out of n_total, why wouldn't you expect to get some meaningful probability of reuse/overlap once your batch grows past size 5 or so (for the sparsest MoE models in common use)? And you only need enough reuse to fill the compute headroom which is quite small on consumer platforms (we won't have huge TOPS numbers for the typical integrated GP…

Yeah, that's what talking out of your butt is literally. "theoretical", no ballparks, ignorant assumptions about expert reuse.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#40

Post by zozbot234 »

lostmsu wrote: post #39
zozbot234 wrote: post #38

Earlier quoted context omitted.

What do you mean by doing the math? If you repeatedly sample n_active experts out of n_total, why wouldn't you expect to get some meaningful probability of reuse/overlap once your batch grows past size 5 or so (for the sparsest MoE models in common use)? And you only need enough reuse to fill the compute headroom which is quite small on consumer platforms (we won't have huge TOPS numbers for the typical integrated GP…

Yeah, that's what talking out of your butt is literally. "theoretical", no ballparks, ignorant assumptions about expert reuse.

There's been some very rough experiments with batching on Apple Silicon (and that's not a highly suitable platform since the compute/thermals bottleneck hits sooner than elsewhere) that seem to be broadly consistent with what I argued, showing as much as 2x total decode throughput with an 8-wide batch. That's substantial in this context.
Post reply on HN