[flagged]
GigaToken: ~1000x faster Language model tokenization
101–110 of 137 posts
Re: GigaToken: ~1000x faster Language model tokenization
#102Earlier quoted context omitted.
You sound like someone who used to write fastruby
Never heard of fast ruby. My point is tokenizing is rarely a bottleneck, as 99% of time is spent in inference. So you speed up 1% of the pipeline by some factor, and the end result is unobservable for a human.
Time to first token is observable by a human, and they’re reporting up to 10% reduction there.
Plus, inference is not the only place tokenization happens. This can make a big difference during development of ML models.
Re: GigaToken: ~1000x faster Language model tokenization
#103[flagged]
Re: GigaToken: ~1000x faster Language model tokenization
#104"AI Use Disclosure: A majority of this code base was crafted by hand without any use of AI (which can be seen from the project's Git history)."
So much for "human programming is obsolete".
Re: GigaToken: ~1000x faster Language model tokenization
#105Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc. Have you considered publishing a rust crate as well? (If not, I volunteer.)
> I hope this code get popular, as it will save tons of electricity, money, CO2, etc.
I don't think tokenization has ever been a meaningful bottleneck. JSON being fast falls into the same bucket much of the time. We spend way more energy on I/O and storage than we do on serialization and tokenization.
If you are concerned with economics and the environment, request batching would make a bigger impact. The most expensive part of this whole thing is GPU underutilization. You can save 50% with OAI right now if you can figure out how to make your workload fit the batch pattern. Do your users always need answers right now or can we afford to wait a few days in some cases? Tool calling doesn't "time out". Wall clock does not exist in the LLM. It took me a while to get used to this.
Re: GigaToken: ~1000x faster Language model tokenization
#106Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep. When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.
Also for embeddings model
Re: GigaToken: ~1000x faster Language model tokenization
#107Earlier quoted context omitted.
Latency isn't performance? Maybe you mean "not throughput-critical"?
but according to Little’s law, if you improve latency, you also improve throughput, right? If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now
It depends on whether or not latency is the constraint of throughput in your circumstances.
Re: GigaToken: ~1000x faster Language model tokenization
#108Congrats, I love performance optimizations!
Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them.
The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
Re: GigaToken: ~1000x faster Language model tokenization
#109Congrats, I love performance optimizations! Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them. The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
> The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.
Haha, yeah, product/executives will surely now see the benefits of optimizations instead of piling new features on top of new features with no cohesive idea about the design or architecture :)
Re: GigaToken: ~1000x faster Language model tokenization
#110[flagged]