Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

github.com

Post reply on HN
Print view View original

51–60 of 79 posts

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#51

Post by gabrielhidasy »

diggan wrote: post #28
matthewolfe wrote: post #26

Earlier quoted context omitted.

Agreed. A former mentor of mine told me a nice way of viewing software development: 1. Make it work. 2. Make it fast. 3. Make it pretty. Transformers & LLMs have been developed to a point where they work quite well. I feel as though we're at a stage where most substantial progress is being made on the performance side.

Heh, seems people I've been learning from been biased away from beauty, as I know that as "Make It Work, Make It Right, Make It Fast".

I always heard the "Make it Right" as "Make it Beautiful", where Right and Beautiful would mean "non-hacky, easily maintainable, easily extendable, well tested, and well documented"

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#52

Post by gabrielhidasy »

stavros wrote: post #34
abybaddi009 wrote: post #33

Earlier quoted context omitted.

What's the difference between make it work and make it right? Aren't they the same thing?

Yeah, if it's not right, it doesn't work.

Depends on your definition of "right" and "work". It could be a big ball of mud that always returns exactly the required response (so it 'works'), but be hellish hard change and very picky about dependencies and environment (so it's not 'right').

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#54

Post by superlopuh »

Can someone familiar with performance of LLMs please tell me how important this is to the overall perf? I'm interested in looking into optimizing tokenizers, and have not yet run the measurements. I would have assumed that the cost is generally dominated by matmuls but am encouraged by the reception of this post in the comments.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#55

Post by stavros »

gabrielhidasy wrote: post #52
stavros wrote: post #34

Earlier quoted context omitted.

Yeah, if it's not right, it doesn't work.

Depends on your definition of "right" and "work". It could be a big ball of mud that always returns exactly the required response (so it 'works'), but be hellish hard change and very picky about dependencies and environment (so it's not 'right').

Nope, it's right, but it's not pretty.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#56

Post by binarymax »

matthewolfe wrote: post #26
npalli wrote: post #5

Kudos, I think (in the short term at least) there is a large amount of perf. optimization to be found by coding parts of the whole AI/ML infrastructure in C++ like this one, not as a rewrite (god no!) but drop in and fix key bottlenecks. Anytime I see someone (seems Chinese engineers are good at this) put something out in C++, good chance some solid engineering tradeoffs have been made and dramatic improvement will b…

Agreed. A former mentor of mine told me a nice way of viewing software development: 1. Make it work. 2. Make it fast. 3. Make it pretty. Transformers & LLMs have been developed to a point where they work quite well. I feel as though we're at a stage where most substantial progress is being made on the performance side.

The Huggingface transformers lib is currently undergoing a refactor to get rid of cruft and make it more extensible, hopefully with some perf gains.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#57

Post by binarymax »

Tiberium wrote: post #50

Can you also compare the performance with https://github.com/huggingface/tokenizers/ ? Would be helpful, since the benchmark in the tiktoken readme seems to be very outdated.

Anecdotally I've always found tiktoken to be far slower than huggingface tokenizers. I'm not sure why, as I haven't dug into tiktoken, but I'm a heavy user of HF's rust tokenizers

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#58

Post by serjester »

superlopuh wrote: post #54

Can someone familiar with performance of LLMs please tell me how important this is to the overall perf? I'm interested in looking into optimizing tokenizers, and have not yet run the measurements. I would have assumed that the cost is generally dominated by matmuls but am encouraged by the reception of this post in the comments.

Tokenizing text is ridiculously small part of the overall computation that goes into serving a request. With that said if you’re doing this on petabytes of data, never hurts to have something faster.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#59

Post by refibrillator »

superlopuh wrote: post #54

Can someone familiar with performance of LLMs please tell me how important this is to the overall perf? I'm interested in looking into optimizing tokenizers, and have not yet run the measurements. I would have assumed that the cost is generally dominated by matmuls but am encouraged by the reception of this post in the comments.

Tokenization is typically done on CPU and is rarely (if ever) a bottleneck for training or inference.

GPU kernels typically dominate in terms of wall clock time, the only exception might be very small models.

Thus the latency of tokenization can essentially be “hidden”, by having the CPU prepare the next batch while the GPU finishes the current batch.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#60

Post by jotux »

matthewolfe wrote: post #26
npalli wrote: post #5

Kudos, I think (in the short term at least) there is a large amount of perf. optimization to be found by coding parts of the whole AI/ML infrastructure in C++ like this one, not as a rewrite (god no!) but drop in and fix key bottlenecks. Anytime I see someone (seems Chinese engineers are good at this) put something out in C++, good chance some solid engineering tradeoffs have been made and dramatic improvement will b…

Agreed. A former mentor of mine told me a nice way of viewing software development: 1. Make it work. 2. Make it fast. 3. Make it pretty. Transformers & LLMs have been developed to a point where they work quite well. I feel as though we're at a stage where most substantial progress is being made on the performance side.

A similar concept dates back to 30BC: https://en.wikipedia.org/wiki/De_architectura

Firmitas, utilitas, venustas - Strong, useful, and beautiful.

Post reply on HN