GigaToken: ~1000x faster Language model tokenization
1–10 of 137 posts
Re: GigaToken: ~1000x faster Language model tokenization
#2What sort of setups do people have that are bounded by the speed of the tokenizer?
Re: GigaToken: ~1000x faster Language model tokenization
#3Interesting :
Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
Re: GigaToken: ~1000x faster Language model tokenization
#4This is really cool, great work!
Re: GigaToken: ~1000x faster Language model tokenization
#5What sort of setups do people have that are bounded by the speed of the tokenizer?
It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches.
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
Re: GigaToken: ~1000x faster Language model tokenization
#6What sort of setups do people have that are bounded by the speed of the tokenizer?
Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!
Re: GigaToken: ~1000x faster Language model tokenization
#7What sort of setups do people have that are bounded by the speed of the tokenizer?
It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches. It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
But are those bounded on the speed of tokenization?
Re: GigaToken: ~1000x faster Language model tokenization
#8What sort of setups do people have that are bounded by the speed of the tokenizer?
Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM.
From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]
I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.
Re: GigaToken: ~1000x faster Language model tokenization
#9What sort of setups do people have that are bounded by the speed of the tokenizer?
Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM. From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) a…
Can't you tokenize in preloading on demand?
Re: GigaToken: ~1000x faster Language model tokenization
#10What sort of setups do people have that are bounded by the speed of the tokenizer?
Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!
That's fair, I just figure there are useful scenarios as well. Apologies if I came off as dismissive!