Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: LLaMA tokenizer that runs in browser

github.com

Post reply on HN
Print view View original

1–10 of 24 posts

Re: Show HN: LLaMA tokenizer that runs in browser

#4

Post by Solvency »

belladoreai wrote: post #2

Hi HN! I was looking for a tokenizer that would accurately(!) count tokens in browser, and I couldn't find one. So I thought "how hard can it be", and here we are 2 weeks later...

For those who would also think the same thing, what're some of the the tldr bulletpoints on why this is more complicated than it'd seem?

Re: Show HN: LLaMA tokenizer that runs in browser

#5

Post by belladoreai »

Solvency wrote: post #4
belladoreai wrote: post #2

Hi HN! I was looking for a tokenizer that would accurately(!) count tokens in browser, and I couldn't find one. So I thought "how hard can it be", and here we are 2 weeks later...

For those who would also think the same thing, what're some of the the tldr bulletpoints on why this is more complicated than it'd seem?

I'll answer with an example.

Consider the input string " grabbed".

If we wanted to map this string to tokens by greedily going from left to right and choosing tokens from the vocabulary with the strategy of minimizing the number of tokens, our algorithm would be very simple. We would end up with the following tokenization: [17229, 2580] == [" grab", "bed"]

Surprisingly, the LLaMA tokenizer does not work this way. It actually finds a "worse" tokenization for this input string: [2646, 1327, 287] == [" gra", "bb", "ed"]

The tokenizer arrives at this 3 token output by applying "merges" in a priority order. For example, this is a merge: [" g", "r"] -> " gr". The trained data contains tens of thousands of these merges. When we apply the merges in the priority order, we end up with 3 tokens.

Now you might be thinking, that's easy, we'll just iterate the list of merges and see if any of them apply. Only problem with that approach is that applying a merge can open up a new opportunity to merge something else that wasn't possible before. This right here is the key thing that makes this problem complicated. We can solve this problem by iterating all possible merges from the beginning after every time we apply a merge. This would produce the correct solution. Only problem is: our algorithm is now very slow and takes minutes to run...

Re: Show HN: LLaMA tokenizer that runs in browser

#6

Post by YetAnotherNick »

belladoreai wrote: post #2

Hi HN! I was looking for a tokenizer that would accurately(!) count tokens in browser, and I couldn't find one. So I thought "how hard can it be", and here we are 2 weeks later...

Great repo, but there was this for openAI which was bit hard to find: https://github.com/cogentapps/chat-with-gpt/blob/main/app/sr....

Not completely sure, but I think it will likely work as it is for llama as both are BPE following same rules.

Re: Show HN: LLaMA tokenizer that runs in browser

#10

Post by pbhjpbhj »

belladoreai wrote: post #5
Solvency wrote: post #4

Earlier quoted context omitted.

For those who would also think the same thing, what're some of the the tldr bulletpoints on why this is more complicated than it'd seem?

I'll answer with an example. Consider the input string " grabbed". If we wanted to map this string to tokens by greedily going from left to right and choosing tokens from the vocabulary with the strategy of minimizing the number of tokens, our algorithm would be very simple. We would end up with the following tokenization: [17229, 2580] == [" grab", "bed"] Surprisingly, the LLaMA tokenizer does not work this way. It…

>Only problem is: //

As a non-programmer this suggests use of a trie? But could you very roughly sketch what you did, or at least give me a keyword or two to lookup? Grazie mille.

Post reply on HN