Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speculative KV coding: losslessly compressing KV cache by up to ~4×

fergusfinn.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#41

Post by lostmsu »

zozbot234 wrote: post #40
lostmsu wrote: post #39

Earlier quoted context omitted.

Yeah, that's what talking out of your butt is literally. "theoretical", no ballparks, ignorant assumptions about expert reuse.

There's been some very rough experiments with batching on Apple Silicon (and that's not a highly suitable platform since the compute/thermals bottleneck hits sooner than elsewhere) that seem to be broadly consistent with what I argued, showing as much as 2x total decode throughput with an 8-wide batch. That's substantial in this context.

Assuming you magically use all 128GiB of xRAM you need to read ~32GiB per token in batched mode. On a good SSD that would be 1/3 tokens per second. Cool, 2x that you can do 2/3 tokens per second. Let's assume you are lucky and can actually do 6/7 tokens per second. That's still an extremely far cry from 20+ tokens per second of 27B before any batching.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#42

Post by fc417fc802 »

lostmsu wrote: post #41
zozbot234 wrote: post #40

Earlier quoted context omitted.

There's been some very rough experiments with batching on Apple Silicon (and that's not a highly suitable platform since the compute/thermals bottleneck hits sooner than elsewhere) that seem to be broadly consistent with what I argued, showing as much as 2x total decode throughput with an 8-wide batch. That's substantial in this context.

Assuming you magically use all 128GiB of xRAM you need to read ~32GiB per token in batched mode. On a good SSD that would be 1/3 tokens per second. Cool, 2x that you can do 2/3 tokens per second. Let's assume you are lucky and can actually do 6/7 tokens per second. That's still an extremely far cry from 20+ tokens per second of 27B before any batching.

I don't understand where your numbers are coming from. Why is there a 20x (40x?) slowdown of tps after batching?

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#43

Post by lostmsu »

fc417fc802 wrote: post #42
lostmsu wrote: post #41

Earlier quoted context omitted.

Assuming you magically use all 128GiB of xRAM you need to read ~32GiB per token in batched mode. On a good SSD that would be 1/3 tokens per second. Cool, 2x that you can do 2/3 tokens per second. Let's assume you are lucky and can actually do 6/7 tokens per second. That's still an extremely far cry from 20+ tokens per second of 27B before any batching.

I don't understand where your numbers are coming from. Why is there a 20x (40x?) slowdown of tps after batching?

Before batching. The slowdown is because the model does not fit the xRAM so experts will have to be read from SSD on every forward pass. That's why it is impractically slow.

Batching could allow you to generate 10 tokens for 10 different conversations at the time, but it also means that you need to load different experts for different tokens, so it does not help as much as it does for dense models.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#44

Post by fc417fc802 »

lostmsu wrote: post #43
fc417fc802 wrote: post #42

Earlier quoted context omitted.

I don't understand where your numbers are coming from. Why is there a 20x (40x?) slowdown of tps after batching?

Before batching. The slowdown is because the model does not fit the xRAM so experts will have to be read from SSD on every forward pass. That's why it is impractically slow. Batching could allow you to generate 10 tokens for 10 different conversations at the time, but it also means that you need to load different experts for different tokens, so it does not help as much as it does for dense models.

But IIUC the point is that each expert gets used for more than just the one token. So yes, the tps of a given thread takes a hit because now you're sometimes going to schedule in unrelated experts and it will have to pause. But overall you're utilizing the hardware much more efficiently and so in aggregate there's a speedup.

On top of that (as previously pointed out by zoz) for a single user running a single overarching task the choice of experts is expected to be highly biased.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#45

Post by lostmsu »

fc417fc802 wrote: post #44
lostmsu wrote: post #43

Earlier quoted context omitted.

Before batching. The slowdown is because the model does not fit the xRAM so experts will have to be read from SSD on every forward pass. That's why it is impractically slow. Batching could allow you to generate 10 tokens for 10 different conversations at the time, but it also means that you need to load different experts for different tokens, so it does not help as much as it does for dense models.

But IIUC the point is that each expert gets used for more than just the one token. So yes, the tps of a given thread takes a hit because now you're sometimes going to schedule in unrelated experts and it will have to pause. But overall you're utilizing the hardware much more efficiently and so in aggregate there's a speedup. On top of that (as previously pointed out by zoz) for a single user running a single overarch…

> the choice of experts is expected to be highly biased

Why? Why do you think that's the case? Part of the training is balancing load between experts.

> so in aggregate there's a speedup.

Yes. 2x. Over theoretical under 1 tok/s

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#46

Post by fc417fc802 »

lostmsu wrote: post #45
fc417fc802 wrote: post #44

Earlier quoted context omitted.

But IIUC the point is that each expert gets used for more than just the one token. So yes, the tps of a given thread takes a hit because now you're sometimes going to schedule in unrelated experts and it will have to pause. But overall you're utilizing the hardware much more efficiently and so in aggregate there's a speedup. On top of that (as previously pointed out by zoz) for a single user running a single overarch…

> the choice of experts is expected to be highly biased Why? Why do you think that's the case? Part of the training is balancing load between experts. > so in aggregate there's a speedup. Yes. 2x. Over theoretical under 1 tok/s

> Why do you think that's the case? Part of the training is balancing load between experts.

That is a fair point. That expectation may have been misplaced on my part. I'm not sufficiently familiar with the details of MoE training.

> The slowdown is because the model does not fit the xRAM so experts will have to be read from SSD on every forward pass.

> 20+ tokens per second of 27B before any batching.

Does the model fit in RAM or not? What is your justification for your stated expectation that the unbatched model will perform 20x faster than the aggregate tps (note, not the single stream tps) of the batched model?

My expectation is that if the unbatched model is 20 tps and batching provides a 2x speedup then each individual stream will be slower but the aggregate throughput should rise to 40 tps. What do you believe me to be missing here?

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#47

Post by zozbot234 »

lostmsu wrote: post #45
fc417fc802 wrote: post #44

Earlier quoted context omitted.

But IIUC the point is that each expert gets used for more than just the one token. So yes, the tps of a given thread takes a hit because now you're sometimes going to schedule in unrelated experts and it will have to pause. But overall you're utilizing the hardware much more efficiently and so in aggregate there's a speedup. On top of that (as previously pointed out by zoz) for a single user running a single overarch…

> the choice of experts is expected to be highly biased Why? Why do you think that's the case? Part of the training is balancing load between experts. > so in aggregate there's a speedup. Yes. 2x. Over theoretical under 1 tok/s

> Why? Why do you think that's the case? Part of the training is balancing load between experts.

The training balances expert choice across the entire scope of the model. Experiments have consistently shown that within a given session or topic (taken in a broad sense) expert choice is biased in a way that's likely to make caching useful and reuse across a user-specific batch realistic.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#48

Post by lostmsu »

fc417fc802 wrote: post #46
lostmsu wrote: post #45

Earlier quoted context omitted.

> the choice of experts is expected to be highly biased Why? Why do you think that's the case? Part of the training is balancing load between experts. > so in aggregate there's a speedup. Yes. 2x. Over theoretical under 1 tok/s

> Why do you think that's the case? Part of the training is balancing load between experts. That is a fair point. That expectation may have been misplaced on my part. I'm not sufficiently familiar with the details of MoE training. > The slowdown is because the model does not fit the xRAM so experts will have to be read from SSD on every forward pass. > 20+ tokens per second of 27B before any batching. Does the model…

27B does, the op was talking about consumer use of DS v4 Flash, that's 160GB.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#49

Post by zozbot234 »

lostmsu wrote: post #48
fc417fc802 wrote: post #46

Earlier quoted context omitted.

> Why do you think that's the case? Part of the training is balancing load between experts. That is a fair point. That expectation may have been misplaced on my part. I'm not sufficiently familiar with the details of MoE training. > The slowdown is because the model does not fit the xRAM so experts will have to be read from SSD on every forward pass. > 20+ tokens per second of 27B before any batching. Does the model…

27B does, the op was talking about consumer use of DS v4 Flash, that's 160GB.

It has been quantized to 80GB (2-bit quantization for experts) with limited degradation. Certainly competitive with a 27B model, and especially useful in a size range where few "native" models exist.

Re: Speculative KV coding: losslessly compressing KV cache by up to ~4×

#50

Post by lostmsu »

zozbot234 wrote: post #49
lostmsu wrote: post #48

Earlier quoted context omitted.

27B does, the op was talking about consumer use of DS v4 Flash, that's 160GB.

It has been quantized to 80GB (2-bit quantization for experts) with limited degradation. Certainly competitive with a 27B model, and especially useful in a size range where few "native" models exist.

> (2-bit quantization for experts) with limited degradation. Certainly competitive with a 27B model

Uh-huh...

Post reply on HN