21–30 of 43 posts
Earlier quoted context omitted.
this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..
yep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.
This is definitely cool and needed, but it might not be so dramatic running 3-5 but quant on a less full GPU.
Does this mean that GPT-4/65b level performance is closer to running on a say a m1/m2 with only 24+ gigabytes of ram?
I guess Flash Attention is more about utilizing memory GPU SRam correctly, where this is more about using the OS/CPU memory better?
I wonder how this compares to Flash Attention ( https://github.com/HazyResearch/flash-attention ), which is the other "memory aware" Attention project I'm aware of. I guess Flash Attention is more about utilizing memory GPU SRam correctly, where this is more about using the OS/CPU memory better?
Now do the same for image classifiers. I tried a few of them, they're just horribly slow. This is pretty outrageous considering the first robust image image classifiers appeared around 2007.
I wonder how this compares to Flash Attention ( https://github.com/HazyResearch/flash-attention ), which is the other "memory aware" Attention project I'm aware of. I guess Flash Attention is more about utilizing memory GPU SRam correctly, where this is more about using the OS/CPU memory better?
Flash attention is just another way to compute exact attention.
This work mainly concerns how to resolve memory fragmentation across different sequences
You still need to compute attention as is once you retrieve the needed key values
Earlier quoted context omitted.
this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..
yep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.