vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention • Hacker News Board

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#31

Post by scv119 » Tue, Jun 20, 2023, 11:48 PM UTC

ipsum2 wrote: post #25

thomasahle wrote: post #24

I wonder how this compares to Flash Attention ( https://github.com/HazyResearch/flash-attention ), which is the other "memory aware" Attention project I'm aware of. I guess Flash Attention is more about utilizing memory GPU SRam correctly, where this is more about using the OS/CPU memory better?

The ideas are orthogonal, and can be used (theoretically) at the same time.

I believe you can slightly change the flash attention kernel to implement the same kernel of this page attention, since both of them work on the key/value cache at block level.

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#32

Post by baobabKoodaa » Tue, Jun 20, 2023, 11:54 PM UTC

zhisbug wrote: post #14

george_123 wrote: post #13

Earlier quoted context omitted.

this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..

yep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.

You mean like, theoretically, in the future? Or you mean today?

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#33

Post by wskwon » Wed, Jun 21, 2023, 12:02 AM UTC

brucethemoose2 wrote: post #22

Reading between the lines, it sounds like some of the speedup comes from VRAM savings on an otherwise close to full GPU? This is definitely cool and needed, but it might not be so dramatic running 3-5 but quant on a less full GPU.

Yes, vLLM focuses on maximizing throughput when the VRAM is fully utilized. Nevertheless, I believe users can still benefit from vLLM even if they don't utilize the memory to its full capacity, because vLLM also includes other optimizations orthogonal to the PagedAttention (e.g., optimized CUDA kernels).

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#34

Post by wskwon » Wed, Jun 21, 2023, 12:13 AM UTC

karmasimida wrote: post #28

thomasahle wrote: post #24

I wonder how this compares to Flash Attention ( https://github.com/HazyResearch/flash-attention ), which is the other "memory aware" Attention project I'm aware of. I guess Flash Attention is more about utilizing memory GPU SRam correctly, where this is more about using the OS/CPU memory better?

I think they are orthogonal. Flash attention is just another way to compute exact attention. This work mainly concerns how to resolve memory fragmentation across different sequences You still need to compute attention as is once you retrieve the needed key values

Thanks for the explanation! I believe the two ideas are basically orthogonal. FlashAttention reduces memory read/writes, while PagedAttention reduces memory waste.

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#35

Post by quickthrower2 » Wed, Jun 21, 2023, 1:40 AM UTC

Now just waiting for a timing attack paper where you can see or guess someone else's conversation that is hosted in the same data center :-).

Or maybe you get typically get dedicated machine time during inference?

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#36

Post by SimFG » Wed, Jun 21, 2023, 3:09 AM UTC

Cool, I prefer the OpenAI-Compatible api. Although this is not very technically difficult, it is really intimate, because it make me feel free to use all ChatGPT applications.

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#37

Post by wskwon » Wed, Jun 21, 2023, 3:54 AM UTC

SimFG wrote: post #36

Cool, I prefer the OpenAI-Compatible api. Although this is not very technically difficult, it is really intimate, because it make me feel free to use all ChatGPT applications.

Thanks! Please try it out and share any feedback you might have.

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#38

Post by e12e » Wed, Jun 21, 2023, 7:31 AM UTC

wskwon wrote: post #20

Solvency wrote: post #5

Semi-related question: this page is full of little charts and diagrams. There are thousands of similar projects/sites/experiment sites with their own charts and diagrams. But it seems like there are always subtle-to-large differences in them that indicate they're made with totally different libraries. Are there just thousands of homebrewn non-standard chart & diagram builders out there? How does one even begin to pic…

We used matplotlib for the performance charts, and used a free website to convert google slides to the animation gifs.

Which "free website"?

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#39

Post by e12e » Wed, Jun 21, 2023, 7:41 AM UTC

wskwon wrote: post #11

kossTKR wrote: post #9

Does this mean that GPT-4/65b level performance is closer to running on a say a m1/m2 with only 24+ gigabytes of ram?

Not really. vLLM optimizes the throughput of your LLM, but does not reduce the minimum required amount of resource to run your model.

But (in theory) - llama.cpp could implement similar approach to paging/memory and see a speedup for 4bit models on cpu?

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#40

Post by jokoon » Wed, Jun 21, 2023, 9:48 AM UTC

ipsum2 wrote: post #27

jokoon wrote: post #15

Now do the same for image classifiers. I tried a few of them, they're just horribly slow. This is pretty outrageous considering the first robust image image classifiers appeared around 2007.

Doesn't work on image classifiers, because there's no KV cache. Also, standard image classifiers can do 100-1000 images/sec without any optimizations.

It's not really fast if makes intense usage of a GPU