Would be cool to see WASM bindings for this here https://github.com/dqbd/tiktoken
Or maybe even your speedups from "b" in the pure js implementation
Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
11–20 of 79 posts
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#12How does this compare to the BPE crate [1]? Its main selling point is support for incrementally re-tokenising text, but it's also faster than tiktoken.
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#13Is there any way we can get local tokenizers for other LLMs? e.g. Gemini only offer a remote API for their tokenizer. Is it proprietary? Could we infer the token mapping somehow efficiently by making lots of calls?
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#14Cool. Would it be possible to eliminate that little vocab format conversion requirement for the vocab I see in the test against tiktoken? It would be nice to have a fully compatible drop in replacement without having to think about details. It also would be nice to have examples that work the other way around: initialize tiktoken as you normally would, including any specialized extension of standard tokenizers, and then use that initialized tokenizer to initialize a new tokendagger and test identity of results.
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#15You know what's also faster to roughly get the amount of tokens? string.length/5
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#16You know what's also faster to roughly get the amount of tokens? string.length/5
[deleted]
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#17What about pairing this with BigBird and Mamba?
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#18Just curious whether it's possible to push any of your performance improvements to tiktoken itself?
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#19[dead]
Re: Show HN: TokenDagger – A tokenizer faster than OpenAI's Tiktoken
#20Is there any way we can get local tokenizers for other LLMs? e.g. Gemini only offer a remote API for their tokenizer. Is it proprietary? Could we infer the token mapping somehow efficiently by making lots of calls?
I thought Gemini used SentencePiece