Is tokenizing really the bottleneck? If we go from 20ms to 15ms does it really matter?
You sound like someone who used to write fastruby
GigaToken: ~1000x faster Language model tokenization
91–100 of 137 posts
Re: GigaToken: ~1000x faster Language model tokenization
#92So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?
The problem with the rest of inference is that changes are not trivially correct or incorrect, as they are with the tokenization layer.
hm, maybe not so trivially correct here. Do I understand correctly that incorrect results can happen as a result of a 42-bit hash collision?
That could happen after less than one MB of input, given the simple one-mul hash.
BTW throughput is measured for a 12 GiB file. Would be interesting to see the throughput for something more like 32 KiB, with cold start (token cache not yet populated).
Re: GigaToken: ~1000x faster Language model tokenization
#93[flagged]
Re: GigaToken: ~1000x faster Language model tokenization
#94[flagged]
The output tokens are identical in either case, but there are quite a few additional settings and formats that huggingface compat mode can generate. In general it also requires inputting Python lists of Python strings.
The Gigatoken API instead prefers input bytes or paths to iterate, and returns an Awkward array by default. Again, no difference in correctness.
Re: GigaToken: ~1000x faster Language model tokenization
#95Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc.
Have you considered publishing a rust crate as well? (If not, I volunteer.)
Re: GigaToken: ~1000x faster Language model tokenization
#96[dead]
Re: GigaToken: ~1000x faster Language model tokenization
#97Is tokenizing really the bottleneck? If we go from 20ms to 15ms does it really matter?
You sound like someone who used to write fastruby
Never heard of fast ruby. My point is tokenizing is rarely a bottleneck, as 99% of time is spent in inference.
So you speed up 1% of the pipeline by some factor, and the end result is unobservable for a human.
Re: GigaToken: ~1000x faster Language model tokenization
#98engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
Except, their benchmarking shows this can speed up time to first token by up to 10%.
Re: GigaToken: ~1000x faster Language model tokenization
#99What sort of setups do people have that are bounded by the speed of the tokenizer?
If you are training an LLM, you need to tokenize the text before it’s trained on. A lot of time this can be done in parallel with the GPU though. I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)
> … only for the run to crash over some minor bug after that
Ah, Python.
Re: GigaToken: ~1000x faster Language model tokenization
#100This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
My understanding is that tokenization is largely serial, so for a large initial prompt it can make up a large chunk of input processing time since after handing it off to the model inference it's (able to be) fully parallel across all tokens.