I've been wondering how to use two spaces as a stop token with the llama models for months. Reading the source of this finally clued me in, "__". Nice. This is significantly easier to comprehend than sentencepiece.
Show HN: LLaMA tokenizer that runs in browser
11–20 of 24 posts
Re: Show HN: LLaMA tokenizer that runs in browser
#12Does anyone know of a chatgpt/gpt-4 tokenizer that can run client-side?
https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
https://platform.openai.com/tokenizer is not for GPT-4 but GPT-3. https://tiktokenizer.vercel.app/ supports GPT-4.
Re: Show HN: LLaMA tokenizer that runs in browser
#13Tokenizers seem to be a massive pain in the neck if you are just calling into an API to use your model. The algorithm itself is non-trivial, and they need pretty sizable data to function: the vocabulary and the merges, which just sit there, using memory. I'm writing https://github.com/ryszard/agency in Go, and while there's a good library for the OpenAI tokenization, if you want a tokenizer for the HF models the best I found was a library calling HF's Rust implementation, which makes it horrible for distribution.
However, at some point I realized that I needed not really the tokens, but the token count, as my most important use was implementing a Token Buffer Memory (trim messages from the beginning in such a way that you never exceed a context size number of tokens). And in order to do that I don't need it to be exactly right, just mostly right, if I am ok with slightly suboptimal efficiency (keeping slightly less tokens than the model supports). So, I took files from Project Gutenberg, and compared the ratio of tokens I get using a proper tokenizer and just calling `strings.Split`, and it seems to be remarkably stable for a given model and language (multiply the length of the result of splitting on spaces by 1.55 for OpenAI and 1.7 for Claude, which leaves a tiny safety margin).
I'm not throwing shade at this project – just being able to call the tokenizer would've saved me a lot of time. But I hope that if I'm wrong about the estimates bring good enough some good person will point out the error of my ways :)
Re: Show HN: LLaMA tokenizer that runs in browser
#14Somewhat tangimential, are there any open source attempts to compete with OpenAI's embeddings?
I know Word2Vec is a thing but I believe that is on a word by word basis, and doesn't capture the semantic meaning of whole sentences and paragraphs.
They charge so little for embeddings I secretly hope they do open source it. Because if for some reason it is stopped, any search functionality or the like that relies upon the API would cease to function
Re: Show HN: LLaMA tokenizer that runs in browser
#15Somewhat tangimential, are there any open source attempts to compete with OpenAI's embeddings? I know Word2Vec is a thing but I believe that is on a word by word basis, and doesn't capture the semantic meaning of whole sentences and paragraphs. They charge so little for embeddings I secretly hope they do open source it. Because if for some reason it is stopped, any search functionality or the like that relies upon th…
https://news.ycombinator.com/item?id=36105660
Models under sentence-transformers are commonly used by people.
You can check this leaderboard, where OpenAI's embeddings are outperformed by open source ones: https://huggingface.co/spaces/mteb/leaderboard
Re: Show HN: LLaMA tokenizer that runs in browser
#16Does anyone know of a chatgpt/gpt-4 tokenizer that can run client-side?
Yes, tiktoken, here's a client side visualizer for it
Re: Show HN: LLaMA tokenizer that runs in browser
#17Tokenizers seem to be a massive pain in the neck if you are just calling into an API to use your model. The algorithm itself is non-trivial, and they need pretty sizable data to function: the vocabulary and the merges, which just sit there, using memory. I'm writing https://github.com/ryszard/agency in Go, and while there's a good library for the OpenAI tokenization, if you want a tokenizer for the HF models the best…
> I get using a proper tokenizer and just calling `strings.Split`, and it seems to be remarkably stable for a given model and language (multiply the length of the result of splitting on spaces by 1.55 for OpenAI and 1.7 for Claude, which leaves a tiny safety margin).
One time I suggested this, got downvoted to hell.
To be fair to the downvoters, I quoted OpenAIs 7 tokens per word(on their tutorial page).
Seems incredibly unrealistic in hindsight, but at the time, things were fresh. Also, I think most people wanted something more robust than a linear calculation.
Re: Show HN: LLaMA tokenizer that runs in browser
#18Does anyone know of a chatgpt/gpt-4 tokenizer that can run client-side?
https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
Thanks for the recommendations. I found the one I needed because of this comment thread.
Re: Show HN: LLaMA tokenizer that runs in browser
#19Does anyone know of a chatgpt/gpt-4 tokenizer that can run client-side?
Thanks for the recommendations everyone.
Re: Show HN: LLaMA tokenizer that runs in browser
#20Earlier quoted context omitted.
For those who would also think the same thing, what're some of the the tldr bulletpoints on why this is more complicated than it'd seem?
I'll answer with an example. Consider the input string " grabbed". If we wanted to map this string to tokens by greedily going from left to right and choosing tokens from the vocabulary with the strategy of minimizing the number of tokens, our algorithm would be very simple. We would end up with the following tokenization: [17229, 2580] == [" grab", "bed"] Surprisingly, the LLaMA tokenizer does not work this way. It…
I take it if you do greedily go from left to right, you end up with different tokenization and the model doesn't work properly because that isn't what it was trained with?
Did the model designers choose this 'difficult' tokenization because it performs better?