Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GigaToken: ~1000x faster Language model tokenization

github.com

Post reply on HN
Print view View original

131–137 of 137 posts

Re: GigaToken: ~1000x faster Language model tokenization

#131

Post by scottcha »

Xalutiono wrote: post #114
scottcha wrote: post #27

Earlier quoted context omitted.

I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.

Whats the prefered LLM runtime to use? vLLM? Tips & Tricks on parameters/settings? What happens at peak? Do people have to wait now? Increase of latency?

We use vllm as it generally has the best ecosystem support. Parameters are largely dependent on what type of requests you are serving (concurrency, input/output ratios, cached hit patterns). We've never had a limitation at the tokenizer step. Limitations at peak tend to manifest more on slower time in vllm doing prefill or decode though we actively try and minimize this.

Re: GigaToken: ~1000x faster Language model tokenization

#132

Post by casey2 »

michaelmior wrote: post #45
luciana1u wrote: post #43

engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable

This depends on what your workflow is. There are use cases for tokenization that don't always involve immediately feeding the text into a model.

There are more usecases, for this class of tokenizer, now that it's 1000x faster as well. RAG being one example.

Re: GigaToken: ~1000x faster Language model tokenization

#133

Post by ubedan »

a_c wrote: post #113
ubedan wrote: post #95

Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc. Have you considered publishing a rust crate as well? (If not, I volunteer.)

Is there any write up regarding the SimdJson Algo? Definitely love to read more of it!

Github: https://github.com/simdjson/simdjson

It showcases an especially ingenious scheme for escaping json strings, as well as the other tokens necessary to parse json.

Daniel Lemire - One of the authors at QCon 2019: https://www.youtube.com/watch?v=wlvKAT7SZIQ

Another Youtube video that explains it: https://www.youtube.com/watch?v=vd9J9PPmAMM

Re: GigaToken: ~1000x faster Language model tokenization

#134

Post by janwas »

marcelroed wrote: post #8
fwip wrote: post #2

What sort of setups do people have that are bounded by the speed of the tokenizer?

Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM. From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) a…

If you are running on large-scale data, have you validated at that scale (comparing results)? From a quick look at the code, it looks like there is a 42-bit hash (computed via single-mul hash function) which can have collisions and thus return the wrong tokens, right?

Re: GigaToken: ~1000x faster Language model tokenization

#135

Post by marcelroed »

ubedan wrote: post #95

Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc. Have you considered publishing a rust crate as well? (If not, I volunteer.)

Yes! I will publish a Rust crate soon. If you have thoughts about how to structure the API I would love to hear them.

Re: GigaToken: ~1000x faster Language model tokenization

#136

Post by rockinghigh »

onlyrealcuzzo wrote: post #14

This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!

For small models, tokenization can reach 1-10% of total inference time.
Post reply on HN