Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GigaToken: ~1000x faster Language model tokenization

github.com

Post reply on HN
Print view View original

71–80 of 137 posts

Re: GigaToken: ~1000x faster Language model tokenization

#71

Post by dataflow »

jaggederest wrote: post #48
scottcha wrote: post #27

Earlier quoted context omitted.

I run an AI platform and we need to tokenize fast and early to make a lot of decisions on the subsequent steps (things like routing, rate limiting and such). Its really important to do this efficiently even though its not a large % of total end to end time for the request.

To concur it's "latency critical", not "performance critical", people often confuse those two - optimize it all, but especially the chained critical path latency!

Latency isn't performance? Maybe you mean "not throughput-critical"?

Re: GigaToken: ~1000x faster Language model tokenization

#73

Post by hansvm »

luciana1u wrote: post #43

engineering effort to make something 1000x faster that accounts for 0.1% of total runtime is the most software developer thing imaginable

1000x improvements unlock qualitatively new capabilities, even when only applied to subcomponents. Moreover, the fact that it's only 0.1% of the runtime is usually [0] an artifact of the entire project being treated that way -- why write this the right way when it won't move the needle on the composite project?

[0] Yes, for LLMs we're closer to the bound than 1000x. Even there though, basic pytorch operations are often 2x slower than simple rewrites, better scheduling algorithms have a history of closer to 5x-10x gains, and all of that supposes you don't have further architectural improvements over time. Moreover, who's to say that legitimately fast tokenization doesn't unlock additional capabilities elsewhere which people have ignored because it was never close to viable?

Re: GigaToken: ~1000x faster Language model tokenization

#75

Post by rockwotj »

dataflow wrote: post #71
jaggederest wrote: post #48

Earlier quoted context omitted.

To concur it's "latency critical", not "performance critical", people often confuse those two - optimize it all, but especially the chained critical path latency!

Latency isn't performance? Maybe you mean "not throughput-critical"?

but according to Little’s law, if you improve latency, you also improve throughput, right?

If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now

Re: GigaToken: ~1000x faster Language model tokenization

#76

Post by cschmidt »

Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas.

And screw all the 0.1% haters on here, this is great stuff.

Re: GigaToken: ~1000x faster Language model tokenization

#77

Post by scheme271 »

rockwotj wrote: post #75
dataflow wrote: post #71

Earlier quoted context omitted.

Latency isn't performance? Maybe you mean "not throughput-critical"?

but according to Little’s law, if you improve latency, you also improve throughput, right? If I have the same number of CPU cores and they all can do their work in half the time they can double the number of requests now

I don't think that's accurate. If tokenization takes say 10ms and the rest of the inference steps take 50 ms then, improving tokenization will improve the time to first token but won't affect throughput much. After the first token, the inference steps effectively hide the tokenization time.

Re: GigaToken: ~1000x faster Language model tokenization

#79

Post by asdf88990 »

cschmidt wrote: post #76

Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas. And screw all the 0.1% haters on here, this is great stuff.

[flagged]

Re: GigaToken: ~1000x faster Language model tokenization

#80

Post by cschmidt »

asdf88990 wrote: post #79
cschmidt wrote: post #76

Can I say this seems to be fantastic work. I cloned your repo earlier today after seeing it on the tokenization discord. I know everyone in the tokenization community wants to absorb the lessons of how you got such a speedup. The caching and replacing the regex for pretokenization seem like generally useful ideas. And screw all the 0.1% haters on here, this is great stuff.

[flagged]

I’m not sure why you think this is ai slop. I work on tokenization research full time. My name is Craig Schmidt and I have a number of papers in the field. This researcher has done some very impressive work and I’m trying to defend him from the HN dismissive hoards.

There is a serious research community on tokenization, and we are quite interested in this work.

Post reply on HN