Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

vllm.ai

Post reply on HN
Print view View original

11–20 of 43 posts

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#11

Post by wskwon »

kossTKR wrote: post #9

Does this mean that GPT-4/65b level performance is closer to running on a say a m1/m2 with only 24+ gigabytes of ram?

Not really. vLLM optimizes the throughput of your LLM, but does not reduce the minimum required amount of resource to run your model.

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#13

Post by george_123 »

bioemerl wrote: post #12

I'm spoiled by 4 bit and unfortunately it doesn't appear to be supposed here so this isn't of much use to me, but it's awesome to see people working on the inference speed side of things regardless.

this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#14

Post by zhisbug »

george_123 wrote: post #13
bioemerl wrote: post #12

I'm spoiled by 4 bit and unfortunately it doesn't appear to be supposed here so this isn't of much use to me, but it's awesome to see people working on the inference speed side of things regardless.

this approach to managing KV cache can work with 4bit. imagine the speedup of pagedattention with quantization..

yep, it is agonistic to 4-bit. You can deploy a 4-bit model and still use vllm + pagedattention to double or even triple your serving throughput.

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#17

Post by kristjansson »

Solvency wrote: post #5

Semi-related question: this page is full of little charts and diagrams. There are thousands of similar projects/sites/experiment sites with their own charts and diagrams. But it seems like there are always subtle-to-large differences in them that indicate they're made with totally different libraries. Are there just thousands of homebrewn non-standard chart & diagram builders out there? How does one even begin to pic…

The color scheme on these implies Google Drawing, but I don't know how they made them into animations - maybe just manually?

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#19

Post by mattnewton »

kristjansson wrote: post #17
Solvency wrote: post #5

Semi-related question: this page is full of little charts and diagrams. There are thousands of similar projects/sites/experiment sites with their own charts and diagrams. But it seems like there are always subtle-to-large differences in them that indicate they're made with totally different libraries. Are there just thousands of homebrewn non-standard chart & diagram builders out there? How does one even begin to pic…

The color scheme on these implies Google Drawing, but I don't know how they made them into animations - maybe just manually?

Google slides I think.

Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention

#20

Post by wskwon »

Solvency wrote: post #5

Semi-related question: this page is full of little charts and diagrams. There are thousands of similar projects/sites/experiment sites with their own charts and diagrams. But it seems like there are always subtle-to-large differences in them that indicate they're made with totally different libraries. Are there just thousands of homebrewn non-standard chart & diagram builders out there? How does one even begin to pic…

We used matplotlib for the performance charts, and used a free website to convert google slides to the animation gifs.
Post reply on HN