[dead]
GigaToken: ~1000x faster Language model tokenization
121–130 of 137 posts
Re: GigaToken: ~1000x faster Language model tokenization
#122[flagged]
Re: GigaToken: ~1000x faster Language model tokenization
#123Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc. Have you considered publishing a rust crate as well? (If not, I volunteer.)
According to Jevons paradox this will likely lead to more electricity used and more CO2 being released to the atmosphere, because it makes it more profitable to build another data center.
Re: GigaToken: ~1000x faster Language model tokenization
#124"AI Use Disclosure: A majority of this code base was crafted by hand without any use of AI (which can be seen from the project's Git history)." So much for "human programming is obsolete".
For context:
In the final stages of the project, AI was used to assist:
Implementing the user-facing API Widening of compatibility, for instance generalizing and porting the pretokenizer implementations to support more tokenizers, less interesting features like padding/truncation/unicode normalization Porting SIMD strategies between AVX512/AVX2/NEON Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy Refactoring and code reuse
Re: GigaToken: ~1000x faster Language model tokenization
#125Earlier quoted context omitted.
In my experience, when adding new features with LLMs, most of those optimizations come automatically. Good models now already follow best practices when implementing, better than junior devs. I wrote a bit about this, I call it "AI slap", lol: https://x.com/XCSme/status/2079115230567686263?s=20
> most of those optimizations come automatically We're clearly thinking of very different "optimizations" here I think :) Do you have any concrete examples of this sort of optimizations you'd get automatically? In my experience, you get what you prompt for, if I don't include to think about performance, they won't think about performance, not sure what model would automatically consider things like that. Most of the…
Not only performance optimizations, but also UI/UX.
If you ask to implement a custom dropdown that does something, it often comes with good spacing, aria-accessible tags, keyboard accessibility, etc. A junior dev wouldn't think of all of those.
Also, it will probably choose the right HTML elements to use for it (i.e. maybe the modern native popover functionality instead of implementing it with custom JS, which would indeed be more efficient and less code).
I am not saying it would add caching by default (even though, it might suggest that), but it's more likely to choose whatever the best options are and to use them as they should, including going around knowing limitations and gotchas.
Re: GigaToken: ~1000x faster Language model tokenization
#126Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas. And screw all the 0.1% haters on here, this is great stuff.
> tokenization discord How can I join this? Sounds interesting
Send me an email (my address is in my profile)
Re: GigaToken: ~1000x faster Language model tokenization
#127Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas. And screw all the 0.1% haters on here, this is great stuff.
>the tokenization discord Could I join this?
Send me an email (my address is in my profile)
Re: GigaToken: ~1000x faster Language model tokenization
#128Re: GigaToken: ~1000x faster Language model tokenization
#129[dead]
Re: GigaToken: ~1000x faster Language model tokenization
#130engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable
I initially had a Rust-based word cloud generator that generates word clouds in high resolution in ~100 milliseconds whereas it would take other generators a couple seconds to do the same thing. Does the world need a super-fast word cloud generator? No. Do I want a super-fast word cloud generator? Yes. I later found enough optimizations to reduce the generation speed all the way down to ~16ms. Do I need a word cloud…
links or it didn't happen ;)