Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GigaToken: ~1000x faster Language model tokenization

github.com

Post reply on HN
Print view View original

111–120 of 137 posts

Re: GigaToken: ~1000x faster Language model tokenization

#111

Post by janalsncm »

antonvs wrote: post #99
janalsncm wrote: post #16

Earlier quoted context omitted.

If you are training an LLM, you need to tokenize the text before it’s trained on. A lot of time this can be done in parallel with the GPU though. I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)

> … only for the run to crash over some minor bug after that Ah, Python.

An example would be setting batch size too high causing OOM which isn’t really a python problem.

Re: GigaToken: ~1000x faster Language model tokenization

#112

Post by lionkor »

XCSme wrote: post #108

Congrats, I love performance optimizations! Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them. The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.

The issue is really with testing! You can do such optimizations but you have to be sure that the result is the same in ALL your use cases, so you need very good test coverage and quite good tests as well.

LLMs can help with the amount of test, and somewhat with the quality, but that's not quite enough for doing heavy optimizations in existing, deployed products, in a normal sprint somewhere.

Re: GigaToken: ~1000x faster Language model tokenization

#113

Post by a_c »

ubedan wrote: post #95

Spectacular... Reminds me of the SimdJson algorithm in terms of jaw dropping nearly unbelievable speeds through creative programming. I hope this code get popular, as it will save tons of electricity, money, CO2, etc. Have you considered publishing a rust crate as well? (If not, I volunteer.)

Is there any write up regarding the SimdJson Algo? Definitely love to read more of it!

Re: GigaToken: ~1000x faster Language model tokenization

#114

Post by Xalutiono »

scottcha wrote: post #27
onlyrealcuzzo wrote: post #14

This is awesome, but tokenization is typically Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!

I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.

Whats the prefered LLM runtime to use? vLLM?

Tips & Tricks on parameters/settings?

What happens at peak? Do people have to wait now? Increase of latency?

Re: GigaToken: ~1000x faster Language model tokenization

#115

Post by Xalutiono »

luciana1u wrote: post #43

engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable

AI now consumes TWh. This is more energy than a lot of countries on the whole planet.

If this increases efficiency, it has real impact. 0.1% is A LOT at scale.

Re: GigaToken: ~1000x faster Language model tokenization

#116

Post by XCSme »

lionkor wrote: post #112
XCSme wrote: post #108

Congrats, I love performance optimizations! Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them. The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.

The issue is really with testing! You can do such optimizations but you have to be sure that the result is the same in ALL your use cases, so you need very good test coverage and quite good tests as well. LLMs can help with the amount of test, and somewhat with the quality, but that's not quite enough for doing heavy optimizations in existing, deployed products, in a normal sprint somewhere.

I think sometimes optimizations are more about using the right data structure/ideas/libraries.

In my case, I recently optimized for https://uxwizz.com the session playback: before, it was saving the entire recording in one row, with updates, and loading it client-side in one chunk. I asked thr AI to, instead, store the recorded chunks in separate rows in db, and when replaying, to stream those chunks as needed. It implemented it perfectly, and everything is much snappier and lighter now.

So, the idea was "store in chunks and stream the chunks", and the LLM does it. Yes, there were a few bugs, but those systems genetally either work or don't. So testing for a 30 minutes after was enough, and now the system is live, and a client actually told me in person yesterday he loved the playback improvements.

I said in another comment somewhere, now knowing how to code isn't important, what's important is that you know what to ask, and what to look for when testing.

Re: GigaToken: ~1000x faster Language model tokenization

#117

Post by XCSme »

embedding-shape wrote: post #109
XCSme wrote: post #108

Congrats, I love performance optimizations! Hardware nowadays is so powerful, but our code so inefficient... I think most libraries/apps could easily be 10x-100x faster if we really try to optimize them. The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly.

> The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly. Haha, yeah, product/executives will surely now see the benefits of optimizations instead of piling new features on top of new features with no cohesive idea about the design or architecture :)

In my experience, when adding new features with LLMs, most of those optimizations come automatically.

Good models now already follow best practices when implementing, better than junior devs.

I wrote a bit about this, I call it "AI slap", lol:

https://x.com/XCSme/status/2079115230567686263?s=20

Re: GigaToken: ~1000x faster Language model tokenization

#118

Post by Cthulhu_ »

alansaber wrote: post #47
luciana1u wrote: post #43

engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable

It's pretty funny but then again, why not if it's as trivial to simplify as it appears

I don't think this was particularly trivial, but I do think that thanks to AI assisted coding there's more capacity for making improvements that "don't seem worth it" at first or when you look at it as a percentage of total.

But look at e.g. Biome; optimizing the formatter didn't seem worth it for a long time because it only took And I'm convinced Biome triggered or was part of a bigger effort to convert JS based tools to native code. This saved time and energy, which in turn allows for faster and / or more feedback loops, which in turn allows faster turnaround cycles for software development (bet it human or LLM assisted), etc. It's a compound effect.

I don't know enough about tokenization or whatever to judge this one, but if it's 1000x as fast as it used to be, there will be less need to try and avoid or minimize tokenization which may lead to new applications.

Re: GigaToken: ~1000x faster Language model tokenization

#119

Post by Cthulhu_ »

Antibabelic wrote: post #104

"AI Use Disclosure: A majority of this code base was crafted by hand without any use of AI (which can be seen from the project's Git history)." So much for "human programming is obsolete".

Nobody except people trying to sell AI are claiming human programming is obsolete though.

Re: GigaToken: ~1000x faster Language model tokenization

#120

Post by embedding-shape »

XCSme wrote: post #117
embedding-shape wrote: post #109

Earlier quoted context omitted.

> The good thing is, that now with AI, we'll probably have the time to implement those optimizations rather quickly. Haha, yeah, product/executives will surely now see the benefits of optimizations instead of piling new features on top of new features with no cohesive idea about the design or architecture :)

In my experience, when adding new features with LLMs, most of those optimizations come automatically. Good models now already follow best practices when implementing, better than junior devs. I wrote a bit about this, I call it "AI slap", lol: https://x.com/XCSme/status/2079115230567686263?s=20

> most of those optimizations come automatically

We're clearly thinking of very different "optimizations" here I think :) Do you have any concrete examples of this sort of optimizations you'd get automatically? In my experience, you get what you prompt for, if I don't include to think about performance, they won't think about performance, not sure what model would automatically consider things like that. Most of the time I use whatever SOTA OpenAI has on maximum reasoning level, fwiw.

Post reply on HN