Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

github.com

Post reply on HN
Print view View original

41–50 of 79 posts

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#41

Post by matthewolfe »

silentsea90 wrote: post #31

"I’m teaching myself LLM internals by re-implementing the stack from first principles." - curious what resources you're using? Any books or courses, or just building it straight up? Great work!

Modal's GPU glossary is a good overview about how GPUs work [0]. Karpathy's LLM overview is a good high level overview on LLMs [1]. 3b1b's video (and subsequent videos) on transformers was excellent at helping me understand the math at a high level [2]. This matrix multiplication optimization worklog helped me understand writing better CUDA (not for beginner intro though) [3].

During this process I also asked ChatGPT a lot of questions.

I'm definitely open to suggestions about "how to learn" with all the new tools we have. I felt this has not been straightforward to figure out.

[0] https://modal.com/gpu-glossary

[1] https://www.youtube.com/watch?v=7xTGNNLPyMI

[2] https://www.youtube.com/watch?v=wjZofJX0v4M

[3] https://siboehm.com/articles/22/CUDA-MMM

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#42

Post by ipsum2 »

npalli wrote: post #5

Kudos, I think (in the short term at least) there is a large amount of perf. optimization to be found by coding parts of the whole AI/ML infrastructure in C++ like this one, not as a rewrite (god no!) but drop in and fix key bottlenecks. Anytime I see someone (seems Chinese engineers are good at this) put something out in C++, good chance some solid engineering tradeoffs have been made and dramatic improvement will b…

Sort of. The key bottlenecks are not in tokenization, but running the actual CUDA kernels. Python actually has very little overhead. (See VLLM, which is primarily in Python). So when people (like deepseek) 'rewrite in C++', they're usually just rewriting CUDA kernels to be more efficient.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#43

Post by kevindamm »

diggan wrote: post #28
matthewolfe wrote: post #26

Earlier quoted context omitted.

Agreed. A former mentor of mine told me a nice way of viewing software development: 1. Make it work. 2. Make it fast. 3. Make it pretty. Transformers & LLMs have been developed to a point where they work quite well. I feel as though we're at a stage where most substantial progress is being made on the performance side.

Heh, seems people I've been learning from been biased away from beauty, as I know that as "Make It Work, Make It Right, Make It Fast".

I've usually heard/said it as

  1. Make it
  2. Make it work
  3. Make it work better
(different circumstances have different nuances about what "better" means, it isn't always performance optimization; some do substitute "faster" for "better" here, but I think it loses generality then).

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#45

Post by matthewolfe »

pama wrote: post #14

Cool. Would it be possible to eliminate that little vocab format conversion requirement for the vocab I see in the test against tiktoken? It would be nice to have a fully compatible drop in replacement without having to think about details. It also would be nice to have examples that work the other way around: initialize tiktoken as you normally would, including any specialized extension of standard tokenizers, and t…

Ah good catch. Updating this right now.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#46

Post by matthewolfe »

matrix2596 wrote: post #44

is is possible for your tokenizer to give different tokenization ever then openai tokenizer? i am asking because there are multiple ways to tokenize the same string?? sry if i am mistaken

Should be the same. Both use Byte-Pair Encoding (BPE) as underlying algo.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#47

Post by bigyabai »

saretup wrote: post #30
npalli wrote: post #5

Kudos, I think (in the short term at least) there is a large amount of perf. optimization to be found by coding parts of the whole AI/ML infrastructure in C++ like this one, not as a rewrite (god no!) but drop in and fix key bottlenecks. Anytime I see someone (seems Chinese engineers are good at this) put something out in C++, good chance some solid engineering tradeoffs have been made and dramatic improvement will b…

And while we’re at it, let’s move away from Python altogether. In the long run it doesn’t make sense just because it’s the language ML engineers are familiar with.

It makes plenty of sense. Python handles strings well, has a great package ecosystem, and is easy to write/learn for non-programmers. It can be easily embedded into a notebook (which is huge for academics) and is technically a "write once run anywhere" platform in theory. It's great.

If you think Python is a bad language for AI integrations, try writing one in a compiled language.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#48

Post by janalsncm »

saretup wrote: post #30
npalli wrote: post #5

Kudos, I think (in the short term at least) there is a large amount of perf. optimization to be found by coding parts of the whole AI/ML infrastructure in C++ like this one, not as a rewrite (god no!) but drop in and fix key bottlenecks. Anytime I see someone (seems Chinese engineers are good at this) put something out in C++, good chance some solid engineering tradeoffs have been made and dramatic improvement will b…

And while we’re at it, let’s move away from Python altogether. In the long run it doesn’t make sense just because it’s the language ML engineers are familiar with.

Most of that is already happening under the hood. A lot of performance-sensitive code is already written in C or cython. For example numpy, scikit learn, pandas. Lots of torch code is either C or CUDA.

ML researchers aren’t using python because they are dumb. They use it because what takes 8 lines in Java can be done with 2 or 3 (including import json) in python for example.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#49

Post by matthewolfe »

pama wrote: post #14

Cool. Would it be possible to eliminate that little vocab format conversion requirement for the vocab I see in the test against tiktoken? It would be nice to have a fully compatible drop in replacement without having to think about details. It also would be nice to have examples that work the other way around: initialize tiktoken as you normally would, including any specialized extension of standard tokenizers, and t…

Alright, 0.1.1 should now be a true drop-in replacement. I'll write up some examples soon.
Post reply on HN