Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Continuous batching to increase LLM inference throughput and reduce p50 latency

anyscale.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Continuous batching to increase LLM inference throughput and reduce p50 latency

#21

Post by jacquesm »

llm_nerd wrote: post #19
sakex wrote: post #3

I was a bit surprised by this: > For example, suppose you prompt with a sentence "What is the capital of California: ", it would take ten forward pass iterations to get back the full response of ["S", "a", "c", "r", “a”, "m", "e", "n", "t", "o"] From the content I have been reading trying to understand LLMs, I thought that the output was a token and not a string of chars. What am I missing here?

A number of submissions lately have "simplified" and presented every character as a token. This has only confused many readers. LLMs use a vocabulary of statistically chosen tokens. GPT 3 vocab, for instance, splits Sacramento into three tokens- Sac - 38318 rament - 15141 o - 78 There is a rule of thumb that about every four letters in English text becomes a token but that's just the average. California is a single t…

Is that because four letters happen to pack into a word on most architectures or is there some other underlying reason? Also Sac and rament together already form a word so it's kind of logical have the final 'o' as a separate token.

Re: Continuous batching to increase LLM inference throughput and reduce p50 latency

#22

Post by daveguy »

jacquesm wrote: post #21
llm_nerd wrote: post #19

Earlier quoted context omitted.

A number of submissions lately have "simplified" and presented every character as a token. This has only confused many readers. LLMs use a vocabulary of statistically chosen tokens. GPT 3 vocab, for instance, splits Sacramento into three tokens- Sac - 38318 rament - 15141 o - 78 There is a rule of thumb that about every four letters in English text becomes a token but that's just the average. California is a single t…

Is that because four letters happen to pack into a word on most architectures or is there some other underlying reason? Also Sac and rament together already form a word so it's kind of logical have the final 'o' as a separate token.

The underlying reason is closer to your second observation. Statistically chosen tokens are most likely to be reusable and composable. Much less to do with hardware architecture. Token vocabulary happens before any quantization where processor word size may play a role.

Re: Continuous batching to increase LLM inference throughput and reduce p50 latency

#23

Post by llm_nerd »

jacquesm wrote: post #21
llm_nerd wrote: post #19

Earlier quoted context omitted.

A number of submissions lately have "simplified" and presented every character as a token. This has only confused many readers. LLMs use a vocabulary of statistically chosen tokens. GPT 3 vocab, for instance, splits Sacramento into three tokens- Sac - 38318 rament - 15141 o - 78 There is a rule of thumb that about every four letters in English text becomes a token but that's just the average. California is a single t…

Is that because four letters happen to pack into a word on most architectures or is there some other underlying reason? Also Sac and rament together already form a word so it's kind of logical have the final 'o' as a separate token.

It's just a function of finding the most efficient solution for encoding a corpus of text into a given vocabulary size. Using something like SentencePiece you can define how big you want your vocabulary to be (the number of discrete tokens), and it will find the best solution of subwords/characters in a sample set.

In the case of GPT-3 the vocabulary has a size of 50,257 tokens. GPT-4 increases that past 100k (see https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a0...).

It's very similar to compression algos, really. Find recurring sets of characters.

Re: Continuous batching to increase LLM inference throughput and reduce p50 latency

#24

Post by evilduck »

ChatGTP wrote: post #15

[flagged]

Your account reliably and almost exclusively posts in AI related topics, or brings AI topics into other threads with predictably negative commentary. I would have assumed this would have been right up your alley.

Re: Continuous batching to increase LLM inference throughput and reduce p50 latency

#26

Post by ChatGTP »

evilduck wrote: post #24
ChatGTP wrote: post #15

[flagged]

Your account reliably and almost exclusively posts in AI related topics, or brings AI topics into other threads with predictably negative commentary. I would have assumed this would have been right up your alley.

Yeah, what's it to ya?
Post reply on HN