Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GigaToken: ~1000x faster Language model tokenization

github.com

Post reply on HN
Print view View original

31–40 of 137 posts

Re: GigaToken: ~1000x faster Language model tokenization

#31

Post by pipsterwo »

boroboro4 wrote: post #28
pipsterwo wrote: post #25

Earlier quoted context omitted.

1/1000 of inference compute is a non-trivial workload at scale. Gartner estimates ~$28B in inference spend for 2026 making this a $28 million dollar per year workload (edit: based on the assumption above) Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...

The issue is it’s cpu compute which is underutilized in gpu clusters anyway, so practically it’s not really 1/1000.

Totally, edited my comment to specify "based on the assumption above." The main takeaway I was going for was 0.1% is not a small number in this context

Re: GigaToken: ~1000x faster Language model tokenization

#33

Post by SOLAR_FIELDS »

vmware508 wrote: post #32

We should just rewrite everything in Rust, especially bloated Python code, and the world would be a better place. ;) Disclosure: I'm a Rust advocate!

1 year ago everyone would have called you insane for suggesting this. Now we all shrug and say yeah maybe we can do this and it’s actually a good idea?

Re: GigaToken: ~1000x faster Language model tokenization

#34

Post by minimaxir »

vmware508 wrote: post #32

We should just rewrite everything in Rust, especially bloated Python code, and the world would be a better place. ;) Disclosure: I'm a Rust advocate!

Both the example libraries compared (tokenizers and tiktoken) are Rust-based with Python bindings. There's just a few levers in Rust that can speed it up even more particularly with LLM assistance as the AI Use Discloure here notes:

> Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy

Re: GigaToken: ~1000x faster Language model tokenization

#36

Post by wren6991 »

marcelroed wrote: post #8
fwip wrote: post #2

What sort of setups do people have that are bounded by the speed of the tokenizer?

Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM. From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) a…

> I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache

Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. That means tokenisation can proceed in parallel with your cache query, and since these caches are distributed in production systems I imagine the query itself could be slow.

I'm not trying to undermine the utility, and this is obviously excellent work. Being able to tokenise faster on the client also seems useful (precise token counts for context pruning heuristics, instead of `chars / 4`), and on a phone your work translates directly to energy savings. I'm just curious about the cache lookup point.

Re: GigaToken: ~1000x faster Language model tokenization

#37

Post by weberer »

vmware508 wrote: post #32

We should just rewrite everything in Rust, especially bloated Python code, and the world would be a better place. ;) Disclosure: I'm a Rust advocate!

We should rewrite all Rust code in Python. Not for any technical reason. I'm just sick of the Rust cult at this point.

Re: GigaToken: ~1000x faster Language model tokenization

#38

Post by noahbp »

onlyrealcuzzo wrote: post #14

This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!

Time to first token, especially for smaller models, can be sharply reduced.

Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.

Re: GigaToken: ~1000x faster Language model tokenization

#39

Post by fastball »

noahbp wrote: post #38
onlyrealcuzzo wrote: post #14

This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!

Time to first token, especially for smaller models, can be sharply reduced. Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.

Tokenization is <0.1% of the inference time for the first token in the same way it is <0.1% for the last.

Re: GigaToken: ~1000x faster Language model tokenization

#40

Post by fastball »

swiftcoder wrote: post #30

So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?

The problem with the rest of inference is that changes are not trivially correct or incorrect, as they are with the tokenization layer.
Post reply on HN