So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?
I'm sure there's been a lot more effort put into the other, more consequential, portions of inference time.
GigaToken: ~1000x faster Language model tokenization
41–50 of 137 posts
Re: GigaToken: ~1000x faster Language model tokenization
#42So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?
the answer is many! This would take hours to write. Full teams and research on nearly every part. So many 'unlocks' coming.
Re: GigaToken: ~1000x faster Language model tokenization
#43engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
Re: GigaToken: ~1000x faster Language model tokenization
#44Practically I would need to wait for hugging face models to adopt this? My harness tokenizer is just an estimate since the model tokenizes on my api calls?
Re: GigaToken: ~1000x faster Language model tokenization
#45engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
This depends on what your workflow is. There are use cases for tokenization that don't always involve immediately feeding the text into a model.
Re: GigaToken: ~1000x faster Language model tokenization
#46engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
"The pursuit of excellence does not need justification."
Re: GigaToken: ~1000x faster Language model tokenization
#47engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
It's pretty funny but then again, why not if it's as trivial to simplify as it appears
Re: GigaToken: ~1000x faster Language model tokenization
#48This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.
To concur it's "latency critical", not "performance critical", people often confuse those two - optimize it all, but especially the chained critical path latency!
Re: GigaToken: ~1000x faster Language model tokenization
#49Earlier quoted context omitted.
Time to first token, especially for smaller models, can be sharply reduced. Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.
Tokenization is <0.1% of the inference time for the first token in the same way it is <0.1% for the last.
Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.
Re: GigaToken: ~1000x faster Language model tokenization
#50Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep.
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.