61–70 of 137 posts
Earlier quoted context omitted.
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
Have you done preliminary numbers on replacing tokenizer on, say, llama-server?
This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.
Earlier quoted context omitted.
> I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. Tha…
It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends. The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count. Again I have no idea wh…
The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.
engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
This is the "GPU driver optimizations don't matter because PC's sit idle at the desktop most of the time" mindset.
This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
sglang_speed [huggingface]: mean=10.31ms median=6.48ms p99=45.98ms rps=96.8
sglang_speed [gigatoken]: mean=10.13ms median=6.54ms p99=45.16ms rps=98.4
input_len= 2048: TTFT mean 30.74 -> 29.05 ms (+5.5% reduction) | median 31.00 -> 28.80 (+7.1%) | p99 33.02 -> 32.02 (+3.0%)
input_len= 8192: TTFT mean 105.20 -> 96.36 ms (+8.4% reduction) | median 103.87 -> 95.49 (+8.1%) | p99 126.88 -> 113.84 (+10.3%)
input_len= 32768: TTFT mean 687.05 -> 633.66 ms (+7.8% reduction) | median 708.14 -> 657.35 (+7.2%) | p99 728.95 -> 678.79 (+6.9%)
Earlier quoted context omitted.
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
Have you done preliminary numbers on replacing tokenizer on, say, llama-server?
So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?
The problem with the rest of inference is that changes are not trivially correct or incorrect, as they are with the tokenization layer.
It will be nicer if the README focuses more on per-core performance.
About the actual algorithm - will something like matching in a perfect hash table help?
engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable