vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
1–10 of 43 posts
Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
#2vLLM has been adopted by LMSYS for serving Vicuna and Chatbot Arena.
Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
#3This is really cool to see.
> Large: Takes up to 1.7GB for a single sequence in LLaMA-13B.
> Dynamic: Its size depends on the sequence length, which is highly variable and unpredictable. As a result, efficiently managing the KV cache presents a significant challenge. We find that existing systems waste 60% – 80% of memory due to fragmentation and over-reservation.
This mentions improvements for throughput which is great, and it mentions memory savings. I'm a bit confused how 80% of the memory could be wasted by the KV cache when the vast majority of the memory is usually holding the model itself?
How much memory savings does this translate to effectively for say a 30B 4bit model?
Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
#4This is really cool to see. > Large: Takes up to 1.7GB for a single sequence in LLaMA-13B. > Dynamic: Its size depends on the sequence length, which is highly variable and unpredictable. As a result, efficiently managing the KV cache presents a significant challenge. We find that existing systems waste 60% – 80% of memory due to fragmentation and over-reservation. This mentions improvements for throughput which is gr…
This really depends on what GPUs you use. If you GPUs has very small amount of memory, vLLM will help more.
vLLM addresses the memory bottleneck for saving KV caches and hence increases the throughput.
Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
#5Semi-related question: this page is full of little charts and diagrams. There are thousands of similar projects/sites/experiment sites with their own charts and diagrams. But it seems like there are always subtle-to-large differences in them that indicate they're made with totally different libraries.
Are there just thousands of homebrewn non-standard chart & diagram builders out there? How does one even begin to pick a standard to whip out quickies like these? Google SEO makes it virtually impossible to get to substance.
Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
#6Is it available an hosted demo?
What are use cases for which open source models are equivalent of GPT 3.5?
Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
#7Is it available an hosted demo? What are use cases for which open source models are equivalent of GPT 3.5?
You can think of LMSYS Vicuna: https://chat.lmsys.org as our hosted demo, as it actually uses vLLM as the backend.
Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
#8Ion Stoica's lab continues to be a powerhouse of innovation. Previous successes of Stoica and his students include (but are certainly not limited to) Apache Spark, Ray, Apache Mesos and Alluxio.
Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
#9Does this mean that GPT-4/65b level performance is closer to running on a say a m1/m2 with only 24+ gigabytes of ram?
Re: vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention
#10Semi-related question: this page is full of little charts and diagrams. There are thousands of similar projects/sites/experiment sites with their own charts and diagrams. But it seems like there are always subtle-to-large differences in them that indicate they're made with totally different libraries. Are there just thousands of homebrewn non-standard chart & diagram builders out there? How does one even begin to pic…
I often see charts produced using matplotlib or plotly - often you can tell based on the colour schemes used. For example, the bar chart at the bottom of this paper looks like it was made with plotly. I think the reason for such variance in the style of charts is largely due to the flexibility frameworks such as matplotlib provide: you can control basically every aspect of a chart and use any number of predefined or custom stylesheets to change the look and feel.