Continuous batching to increase LLM inference throughput and reduce p50 latency
1–10 of 28 posts
Re: Continuous batching to increase LLM inference throughput and reduce p50 latency
#2As this article is from some weeks ago and Huggingface has now implemented Paged Attention in text-generation-inference[1], I would assume the benchmark results would be quite different if done today. Would be very interesting to see more recent benchmarks if anyone has done any!
[1] https://github.com/huggingface/text-generation-inference/iss...
Re: Continuous batching to increase LLM inference throughput and reduce p50 latency
#3I was a bit surprised by this:
> For example, suppose you prompt with a sentence "What is the capital of California: ", it would take ten forward pass iterations to get back the full response of ["S", "a", "c", "r", “a”, "m", "e", "n", "t", "o"]
From the content I have been reading trying to understand LLMs, I thought that the output was a token and not a string of chars. What am I missing here?
Re: Continuous batching to increase LLM inference throughput and reduce p50 latency
#4I was a bit surprised by this: > For example, suppose you prompt with a sentence "What is the capital of California: ", it would take ten forward pass iterations to get back the full response of ["S", "a", "c", "r", “a”, "m", "e", "n", "t", "o"] From the content I have been reading trying to understand LLMs, I thought that the output was a token and not a string of chars. What am I missing here?
I suspect they’re simplifying things
Re: Continuous batching to increase LLM inference throughput and reduce p50 latency
#5I was a bit surprised by this: > For example, suppose you prompt with a sentence "What is the capital of California: ", it would take ten forward pass iterations to get back the full response of ["S", "a", "c", "r", “a”, "m", "e", "n", "t", "o"] From the content I have been reading trying to understand LLMs, I thought that the output was a token and not a string of chars. What am I missing here?
The tokens are chosen by a method called BPE, and there are single letter tokens. IOW you can encode the same text many ways. That said, this is probably just shown like this for illustrative purposes.
Re: Continuous batching to increase LLM inference throughput and reduce p50 latency
#6I was a bit surprised by this: > For example, suppose you prompt with a sentence "What is the capital of California: ", it would take ten forward pass iterations to get back the full response of ["S", "a", "c", "r", “a”, "m", "e", "n", "t", "o"] From the content I have been reading trying to understand LLMs, I thought that the output was a token and not a string of chars. What am I missing here?
It's mentioned in the article: "This example simplifies things a little bit because in actuality tokens do not map 1:1 to ASCII characters (a popular token encoding technique is Byte-Pair Encoding which is beyond the scope of this blog post), but the iterative nature of generation is the same regardless of how you tokenize your sequences."
It's an extremization that still is true for character-based models.
Re: Continuous batching to increase LLM inference throughput and reduce p50 latency
#7I was a bit surprised by this: > For example, suppose you prompt with a sentence "What is the capital of California: ", it would take ten forward pass iterations to get back the full response of ["S", "a", "c", "r", “a”, "m", "e", "n", "t", "o"] From the content I have been reading trying to understand LLMs, I thought that the output was a token and not a string of chars. What am I missing here?
You need to use a decoder to convert the response to a sentence.
Example: https://huggingface.co/docs/tokenizers/api/decoders
Re: Continuous batching to increase LLM inference throughput and reduce p50 latency
#8I was a bit surprised by this: > For example, suppose you prompt with a sentence "What is the capital of California: ", it would take ten forward pass iterations to get back the full response of ["S", "a", "c", "r", “a”, "m", "e", "n", "t", "o"] From the content I have been reading trying to understand LLMs, I thought that the output was a token and not a string of chars. What am I missing here?
[deleted]
Re: Continuous batching to increase LLM inference throughput and reduce p50 latency
#9As this article is from some weeks ago and Huggingface has now implemented Paged Attention in text-generation-inference[1], I would assume the benchmark results would be quite different if done today. Would be very interesting to see more recent benchmarks if anyone has done any! [1] https://github.com/huggingface/text-generation-inference/iss...
As always the LLM hamster wheel is moving too fast. I’d love to matrix download half this stuff (I still enjoy learning so classic-learn the other half)
Re: Continuous batching to increase LLM inference throughput and reduce p50 latency
#10As this article is from some weeks ago and Huggingface has now implemented Paged Attention in text-generation-inference[1], I would assume the benchmark results would be quite different if done today. Would be very interesting to see more recent benchmarks if anyone has done any! [1] https://github.com/huggingface/text-generation-inference/iss...
[deleted]