[flagged]
Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
61–70 of 79 posts
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#62Can someone familiar with performance of LLMs please tell me how important this is to the overall perf? I'm interested in looking into optimizing tokenizers, and have not yet run the measurements. I would have assumed that the cost is generally dominated by matmuls but am encouraged by the reception of this post in the comments.
Tokenizing text is ridiculously small part of the overall computation that goes into serving a request. With that said if you’re doing this on petabytes of data, never hurts to have something faster.
A language that isn’t memory-safe can definitely hurt. AI needs more security, not less.
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#63Can someone familiar with performance of LLMs please tell me how important this is to the overall perf? I'm interested in looking into optimizing tokenizers, and have not yet run the measurements. I would have assumed that the cost is generally dominated by matmuls but am encouraged by the reception of this post in the comments.
Tokenization performance is complicated, but your guidepost is that the institutions with the resources and talent to do so choose to write extremely fast tokenizers: sentencepiece and tiktoken both pay dearly in complexity (particularly complexity of deployment because now you've got another axis of architecture-specific build/bundle/dylib to manage in addition to whatever your accelerator burden always was: its now aarch64 cross x86_64 cross CUDA capability...)
Sometimes it can overlap with accelerator issue, but pros look at flame graphs: a CPU core running the AVX lanes hard isn't keeping the bus fed, million things. People pre-tokenize big runs all the time.
I don't know why this thread is full of "nothing to see here", this obliterates the SOTA from the money is no object status quo: I'd like to think better of the community than the obvious which is that C++ is threatening a modest mindshare comeback against a Rust narrative that's already under pressure from the explosion of interest in Zig. Maybe there's a better reason.
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#64Very cool. We use Tiktoken and I'd love to see the performance impact. Pretty great decision to make it drop-in compatible.
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#65Now that byte-patch-level embeddings are discovered?
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#66Kudos, I think (in the short term at least) there is a large amount of perf. optimization to be found by coding parts of the whole AI/ML infrastructure in C++ like this one, not as a rewrite (god no!) but drop in and fix key bottlenecks. Anytime I see someone (seems Chinese engineers are good at this) put something out in C++, good chance some solid engineering tradeoffs have been made and dramatic improvement will b…
It looks like TikToken is written in Rust (https://github.com/openai/tiktoken/tree/main/src), are the gains here actually from porting to C++?
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#67I'm relieved to see that its not written in rust
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#68Earlier quoted context omitted.
Agreed. A former mentor of mine told me a nice way of viewing software development: 1. Make it work. 2. Make it fast. 3. Make it pretty. Transformers & LLMs have been developed to a point where they work quite well. I feel as though we're at a stage where most substantial progress is being made on the performance side.
Heh, seems people I've been learning from been biased away from beauty, as I know that as "Make It Work, Make It Right, Make It Fast".
Fair chance I'm remembering it wrong :D
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#69I'm relieved to see that its not written in rust
haha, I thought about it.
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#70Can someone familiar with performance of LLMs please tell me how important this is to the overall perf? I'm interested in looking into optimizing tokenizers, and have not yet run the measurements. I would have assumed that the cost is generally dominated by matmuls but am encouraged by the reception of this post in the comments.
To echo the other replies, the tokenizer is definitely not the bottleneck. It just happens to be the first step in inference, so it's what I did first.