Show HN: LLaMA tokenizer that runs in browser

This sounds reasonable to me. You might also want to consider estimates based on the number of characters. And you also need a fallback for what to do when the user inputs some weird input that doesn't fall inside your safety margin, but instead causes OpenAI API to return an error (maybe in that case you aggressively trim the input and retry?)

> I take it if you do greedily go from left to right, you end up with different tokenization and the model doesn't work properly because that isn't what it was trained with?

The way that the LLaMA tokenizer works is, a word is tokenized one way within a certain context, and a different way within a different context. If you go greedily from left to right, you will find mostly the same tokenizations for words, but occasionally the tokenization will be different. It will still be a legitimate tokenization for the word, though, so the model will not spew nonsense if you feed these tokens into it. I assume the outputs of the model would be slightly suboptimal, but the difference might be so small that nobody notices anything.

Anyway, I don't expect anyone to use this library in a manner where they will turn text to tokens with this library, and then feed the tokens to the model. I expect people to use this library to count tokens, dynamically iterate on the prompt to fit within the token limit, and then eventually send raw text to the model, which performs the tokenization by itself. If we think about this use case, and we consider what would happen if we just greedily tokenized from left to right, it would be mostly fine, because our token counts would be pretty close to the real token count. But occasionally it wouldn't be fine: occasionally our greedy tokenizer would say that we are below the token limit, when actually we're beyond the token limit. In this case the model would either return an error, or it would automatically trim some part of the prompt. If the prompt is automatically trimmed, it might be trimmed from the wrong place. For example, oobabooga trims the prompt from the beginning, when you typically would want to trim the prompt from the middle.

> Did the model designers choose this 'difficult' tokenization because it performs better?

I'm not sure. LLaMA is using a SentencePiece BPE tokenizer, and the most-often touted benefit of this tokenizer is that it can fluently deal with multiple different languages. So it's possible that they chose this to support multiple different languages, not to maximize the quality of output in the English language.