Very interesting project! Are there benchmarks for the "compatibility mode" or are all the numbers for the Gigatoken API?
GigaToken: ~1000x faster Language model tokenization
21–30 of 137 posts
Re: GigaToken: ~1000x faster Language model tokenization
#22Very interesting project! Are there benchmarks for the "compatibility mode" or are all the numbers for the Gigatoken API?
Numbers are for the Gigatoken API, but compatibility mode just means eating a bunch of Python overhead (creating lists, reading strings to bytes). You can expect a modest ~200-300x speedup with compatibility mode depending on how you use it.
Re: GigaToken: ~1000x faster Language model tokenization
#23Very interesting project! Are there benchmarks for the "compatibility mode" or are all the numbers for the Gigatoken API?
I can add some benchmarks for compatibility mode in the future. I have a little more juice to squeeze out of the Python interop though, so not quite ready for it yet.
Re: GigaToken: ~1000x faster Language model tokenization
#24This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
Always good to make it 0.001%
Re: GigaToken: ~1000x faster Language model tokenization
#25This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above)
Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...
Re: GigaToken: ~1000x faster Language model tokenization
#26Quality software here.
Re: GigaToken: ~1000x faster Language model tokenization
#27This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.
Re: GigaToken: ~1000x faster Language model tokenization
#28This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above) Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...
The issue is it’s cpu compute which is underutilized in gpu clusters anyway, so practically it’s not really 1/1000.
Re: GigaToken: ~1000x faster Language model tokenization
#29This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
[dead]
Re: GigaToken: ~1000x faster Language model tokenization
#30So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?