Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

github.com

Post reply on HN
Print view View original

71–79 of 79 posts

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#71

Post by mindcrime »

diggan wrote: post #28
matthewolfe wrote: post #26

Earlier quoted context omitted.

Agreed. A former mentor of mine told me a nice way of viewing software development: 1. Make it work. 2. Make it fast. 3. Make it pretty. Transformers & LLMs have been developed to a point where they work quite well. I feel as though we're at a stage where most substantial progress is being made on the performance side.

Heh, seems people I've been learning from been biased away from beauty, as I know that as "Make It Work, Make It Right, Make It Fast".

I've always heard it (and said it) as:

  1. Make it work
  2. Make it correct
  3. Make it fast

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#72

Post by mdaniel »

bigyabai wrote: post #47
saretup wrote: post #30

Earlier quoted context omitted.

And while we’re at it, let’s move away from Python altogether. In the long run it doesn’t make sense just because it’s the language ML engineers are familiar with.

It makes plenty of sense. Python handles strings well, has a great package ecosystem, and is easy to write/learn for non-programmers. It can be easily embedded into a notebook (which is huge for academics) and is technically a "write once run anywhere" platform in theory. It's great. If you think Python is a bad language for AI integrations, try writing one in a compiled language.

> has a great package ecosystem

So great there are 8 of them. 800% better than all the rest!

> If you think Python is a bad language for AI integrations, try writing one in a compiled language.

I'll take this challenge, all day, every day, so long as I and the hypothetical 'move fast and break things' have equal "must run in prod" and "must be understandable by some other human" qualifiers

What type is `array`? Don't worry your pretty head about it, feed it whatever type you want and let Sentry's TypeError sort it out https://github.com/openai/whisper/blob/v20250625/whisper/aud...> Oh, sorry, and you wanted to know what `pad_or_trim` returns? Well that's just, like, your opinion man

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#73

Post by bigyabai »

mdaniel wrote: post #72
bigyabai wrote: post #47

Earlier quoted context omitted.

It makes plenty of sense. Python handles strings well, has a great package ecosystem, and is easy to write/learn for non-programmers. It can be easily embedded into a notebook (which is huge for academics) and is technically a "write once run anywhere" platform in theory. It's great. If you think Python is a bad language for AI integrations, try writing one in a compiled language.

> has a great package ecosystem So great there are 8 of them. 800% better than all the rest! > If you think Python is a bad language for AI integrations, try writing one in a compiled language. I'll take this challenge, all day, every day, so long as I and the hypothetical 'move fast and break things' have equal "must run in prod" and "must be understandable by some other human" qualifiers What type is `array`? Don't…

Tracks with me, I don't like using Python for real programming. Try explaining any of your "Python sucks" catechisms to a second-year statistics student though. If you'd rather teach them C++, be my guest. If you want to make them indebted to proprietary infra like Mojo or CUDA, knock yourself out.

I'm still teaching them Python.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#74

Post by fhub »

notatallshaw wrote: post #66
npalli wrote: post #5

Kudos, I think (in the short term at least) there is a large amount of perf. optimization to be found by coding parts of the whole AI/ML infrastructure in C++ like this one, not as a rewrite (god no!) but drop in and fix key bottlenecks. Anytime I see someone (seems Chinese engineers are good at this) put something out in C++, good chance some solid engineering tradeoffs have been made and dramatic improvement will b…

It looks like TikToken is written in Rust ( https://github.com/openai/tiktoken/tree/main/src ), are the gains here actually from porting to C++?

From the post

Profiling TikToken’s Python/Rust implementation showed a lot of time was spent doing regex matching. Most of my perf gains come from a) using a faster jit-compiled regex engine; and b) simplifying the algorithm to forego regex matching special tokens at all.

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#75

Post by acosmism »

kevindamm wrote: post #43
diggan wrote: post #28

Earlier quoted context omitted.

Heh, seems people I've been learning from been biased away from beauty, as I know that as "Make It Work, Make It Right, Make It Fast".

I've usually heard/said it as 1. Make it 2. Make it work 3. Make it work better (different circumstances have different nuances about what "better" means, it isn't always performance optimization; some do substitute "faster" for "better" here, but I think it loses generality then).

i like this version best

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#77

Post by 22c »

kevmo314 wrote: post #22

Nice work! I tried something similar a while back ago: https://github.com/kevmo314/tokie The takeaway I also found was that the running cost was really dominated by pretokenization (the regex). It's cool to see that you found a faster way to run the regex, but have you tried comparing the performance of just swapping out the regex engine and leaving the actual BPE to tiktoken? I wonder if that is upstreamable?

There is at least some awareness already when it comes to the performance of the regex engine:

https://github.com/openai/tiktoken/blob/main/src/lib.rs#L95-...

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#78

Post by justinhj »

I've been playing with tokenization too. Starting from Kaparthy's Python minbpe I set myself the task of training a tokenizer on wikitext (500mb) in a reasonable time. I got the C++ version down to about 50 minutes compared to the original Python code (estimated) several months.

Haven't really spent much time looking at encode and decode but I plan to incorporate these regex modifications when I do!

https://github.com/justinhj/minbpe-cc

Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken

#79

Post by anonymoushn »

matthewolfe wrote: post #9
konsalexee wrote: post #8

> simplifying the algorithm to forego regex matching special tokens at all Does that mean there could be cases with less quality in terms of tokenization?

The output should be identical, assuming no bugs. The Tiktoken implementation takes a collection of all special tokens upon initialization and compiles them into a regex by joining them with `|` [0]. Then the actual encoding process checks for matches on this expression. Models like Llama 4 define a list of 1,135 special tokens. Notably, 1,115 of those are "reserved" special tokens! So this yields a huge regexp of sp…

Isn't this incorrect? If the user doesn't specify what to do with almost all of the special tokens, you still must detect them so you can raise an error.
Post reply on HN