Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

ndingwall.github.io

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

#21

Post by Lerc »

mrkramer wrote: post #16

So LLMs hallucinate by design which is of course bad for any serious work but then again I was thinking in which situations would hallucinations be actually useful....let's say poetry writing, lyrics writing etc., the crazier the better or for generating business ideas, "crazy ideas succeed" type logic.

While training something on TinyStories one of the early stage outputs was

"Once upon a time, in a small town, there was a huge town"

Which I thought was the brilliant start for an interesting story. Just that slight deviation from expectation provides a excellent scaffold for your imagination to take hold.

Of course the rest of the generation was utter rubbish, but if you could on-demand deviate from standard at key points while keeping the majority internally consistent then it would be great.

Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

#22

Post by anonymoushn »

pama wrote: post #15
anonymoushn wrote: post #14

Earlier quoted context omitted.

https://x.com/alexjc/status/1881410039639863622 I agree that it would be more useful to compare vocabs of identical size.

Thanks! It seems to me that the performance gain here was due to a smaller vocab size. This type of change is almost guaranteed to backfire for larger models and larger datasets / lower loss requirements, so it probably is not very useful. Generally the historical trend has been to use larger vocabularies as the models got larger themselves.

Well, he says he achieved some downstream win on the same size, but it didn't translate into a win in perplexity, so he tried to do something else. Like I said, it's unfortunate.

Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

#23

Post by mrkramer »

Lerc wrote: post #21
mrkramer wrote: post #16

So LLMs hallucinate by design which is of course bad for any serious work but then again I was thinking in which situations would hallucinations be actually useful....let's say poetry writing, lyrics writing etc., the crazier the better or for generating business ideas, "crazy ideas succeed" type logic.

While training something on TinyStories one of the early stage outputs was "Once upon a time, in a small town, there was a huge town" Which I thought was the brilliant start for an interesting story. Just that slight deviation from expectation provides a excellent scaffold for your imagination to take hold. Of course the rest of the generation was utter rubbish, but if you could on-demand deviate from standard at key…

Poetry is full of figures of speech which are words and phrases that intentionally differ from common logic and common knowledge so LLMs can actually be probabilistic and accidental generators of figures of speech. Every nonsense that LLMs generate can be interpreted as a protentional metaphor or an oxymoron. The sentence that you mentioned sounds like a badass metaphor to me "Once upon a time, in a small town, there was a huge town" or in another words: town might be small in size but it has soul.

Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

#24

Post by yuvalpinter »

mdaniel wrote: post #10
kevmo314 wrote: post #9

Earlier quoted context omitted.

Unfortunately the choice of tokenizer is baked into the model. If you want to innovate on the tokenizer, you have to train a whole base model yourself to prove it's better which makes the barrier for innovation pretty high.

Apologies if this is a dumb question, but is there no "hello world"-ish sandbox for testing this theory? I can very easily imagine that trying to go head-to-head with R1 or such is going to be a boatload of GPU, but for just testing tokenizer head-to-head isn't there a smaller sized one that can be used in a bake off?

We've been working on this problem for quite some time in my lab. We released a benchmark piling together several "intrinsic evaluations" that don't require model training. We're currently investigating correlations between performance on this benchmark and on downstream tasks. Here it is - https://github.com/MeLeLBGU/tokenizers_intrinsic_benchmark - where there's a link to the paper we introduced it in, used for checking how inference schemes work together with various token vocabs. It's for English, but cited work and citing work have some of these for other languages as well.

Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

#25

Post by yuvalpinter »

YossarianFrPrez wrote: post #20
janalsncm wrote: post #13

Interesting recent research from Meta going in what I would call the opposite direction from an interpretability standpoint: https://arxiv.org/abs/2412.09871 Rather than splitting based on a predefined list you can check with a text editor, BLTs use a neural net to tokenize. So it will be pretty hard to debug.

This is a cool paper! Basically, prior to feeding text to the tokenizer people have split the text on whitespaces. But whitespaces aren't exactly meaningful separators. By getting rid of this restriction as the tokenizer is 'learning', some of the tokens end up being 'by the way' or 'in the the long run.' The researchers find that this makes the model much more efficient.

This can also be done within the tokenization framework, see our work here: https://arxiv.org/abs/2504.00178

Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

#26

Post by yuvalpinter »

anonymoushn wrote: post #8
pbronez wrote: post #6

Is there a more current post with similar information? I’d love to see how contemporary tokenizers have improved on these algorithms.

They have not.

A few years ago we released SaGe, which is a contextual tokenizer, meaning that it builds a vocab that's fit for an LM use case because tokens are selected to appear within as clear a context set as possible in a corpus. https://github.com/MeLeLBGU/SaGe It does better than both BPE and UnigramLM in several benchmarks. https://aclanthology.org/2024.acl-short.73/

Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

#27

Post by anonymoushn »

yuvalpinter wrote: post #26
anonymoushn wrote: post #8

Earlier quoted context omitted.

They have not.

A few years ago we released SaGe, which is a contextual tokenizer, meaning that it builds a vocab that's fit for an LM use case because tokens are selected to appear within as clear a context set as possible in a corpus. https://github.com/MeLeLBGU/SaGe It does better than both BPE and UnigramLM in several benchmarks. https://aclanthology.org/2024.acl-short.73/

Cool, I can have a look. By the way, I've filed a PR against the pathpiece repository correcting an error in the docs.

I have read this paper before, and now that I have looked more closely I am a bit disappointed that the precise method of construction of the large initial vocabularies used for vocabulary learners that begin with large initial vocabularies in the experiment is not specified. I also notice that the same error I have proposed correcting in the docs is present in the paper in section 3.1.

Additional edit: Alright, I see that A.2.1 specifies that the "n-gram" initial vocab condition means the most frequent 2^18 n-grams for n in 1..L. This seems like not very many, and not weighting n-grams by "potential CTC reduction" (=length-1) for inclusion in this initial vocab seems to create a bias against long n-grams. I'm also sort of wary of these greedy learners, because many candidates for inclusion in the vocab are interdependent with other candidates (the marginal loss or gain of a candidate may be near 0 because of the presence of some other token which may not actually be optimal to include, or there may be groups of included tokens which cause other better groups to have near 0 marginal loss or gain), so I would really want to avoid being too strongly path-dependent in the learner. If I may hand wave a great deal, I think this sort of issue also arises if your objective function is not CTC.

Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

#28

Post by anonymoushn »

anonymoushn wrote: post #22
pama wrote: post #15

Earlier quoted context omitted.

Thanks! It seems to me that the performance gain here was due to a smaller vocab size. This type of change is almost guaranteed to backfire for larger models and larger datasets / lower loss requirements, so it probably is not very useful. Generally the historical trend has been to use larger vocabularies as the models got larger themselves.

Well, he says he achieved some downstream win on the same size, but it didn't translate into a win in perplexity, so he tried to do something else. Like I said, it's unfortunate.

I actually wonder if he could just claim a win by calculating validation set BPB for both equally-sized vocabs instead of targeting the same level of perplexity as in the speedrun finish line lol

Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

#29

Post by cschmidt »

anonymoushn wrote: post #2

How does SentencePiece choose the initial vocabulary, which is trimmed down to determine the final vocabulary which has these desirable properties?

It appears to be the top n-grams scored by the product of frequency and length. Including the frequency weighting is a bit nonstandard among ablative methods.

See line 233: https://github.com/google/sentencepiece/blob/master/src/unig...

I would suspect the n-gram counts don't cross pre-token boundaries, but I don't have time to find that in the code right now.

Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

#30

Post by cschmidt »

anonymoushn wrote: post #27
yuvalpinter wrote: post #26

Earlier quoted context omitted.

A few years ago we released SaGe, which is a contextual tokenizer, meaning that it builds a vocab that's fit for an LM use case because tokens are selected to appear within as clear a context set as possible in a corpus. https://github.com/MeLeLBGU/SaGe It does better than both BPE and UnigramLM in several benchmarks. https://aclanthology.org/2024.acl-short.73/

Cool, I can have a look. By the way, I've filed a PR against the pathpiece repository correcting an error in the docs. I have read this paper before, and now that I have looked more closely I am a bit disappointed that the precise method of construction of the large initial vocabularies used for vocabulary learners that begin with large initial vocabularies in the experiment is not specified. I also notice that the s…

Somehow I didn't get any notifications of your PR. Sorry about that. I'll take a look.
Post reply on HN