Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
ndingwall.github.io
Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
1–10 of 40 posts
Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
#2How does SentencePiece choose the initial vocabulary, which is trimmed down to determine the final vocabulary which has these desirable properties?
Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
#3The vibe of AI discussions was so much different with today's (or at least for the past 2.5 years at least).
It's quite surreal how things moved so fast
Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
#4How does SentencePiece choose the initial vocabulary, which is trimmed down to determine the final vocabulary which has these desirable properties?
Just a minor nit: SentencePiece is a library, not a tokenization algorithm. It implements two tokenization algorithms, Unigram and BPE.
BPE builds vocabularies from the base up so I assume you are talking about Unigram which starts with a big vocabulary and trims it.
The details of UnigramLM are here https://arxiv.org/pdf/1804.10959, and the part about vocabulary seeding is Section 3.2.
Basically, it just selects all substrings that appear in the corpus up to a certain length (and then maybe trims it a little by discarding rare substrings or something to reduce the initial size a bit and make things faster).
Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
#5It's always seemed like such low hanging fruit, so much semantic information just squandered in the thirst for larger models.
Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
#6Is there a more current post with similar information? I’d love to see how contemporary tokenizers have improved on these algorithms.
Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
#7How does SentencePiece choose the initial vocabulary, which is trimmed down to determine the final vocabulary which has these desirable properties?
Just a minor nit: SentencePiece is a library, not a tokenization algorithm. It implements two tokenization algorithms, Unigram and BPE. BPE builds vocabularies from the base up so I assume you are talking about Unigram which starts with a big vocabulary and trims it. The details of UnigramLM are here https://arxiv.org/pdf/1804.10959 , and the part about vocabulary seeding is Section 3.2. Basically, it just selects al…
If the library has two vocabulary learners, only one of which does the described thing, then isn't it unambiguous which implementation within the library the question refers to? And wouldn't it be ambiguous to instead say "how does Unigram do it" without referring to any particular implementation?
Anyway, the paper says "Frequent substrings can be enumerated in O(T) time and O(20T) space with the Enhanced Suffix Array algorithm (Nong et al., 2009)", which is hilariously underspecified, at least in part because a suffix array algorithm isn't a top-k algorithm.
Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
#8Is there a more current post with similar information? I’d love to see how contemporary tokenizers have improved on these algorithms.
They have not.
Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
#9It's always seemed like such low hanging fruit, so much semantic information just squandered in the thirst for larger models.
Unfortunately the choice of tokenizer is baked into the model. If you want to innovate on the tokenizer, you have to train a whole base model yourself to prove it's better which makes the barrier for innovation pretty high.
Re: Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)
#10It's always seemed like such low hanging fruit, so much semantic information just squandered in the thirst for larger models.
Unfortunately the choice of tokenizer is baked into the model. If you want to innovate on the tokenizer, you have to train a whole base model yourself to prove it's better which makes the barrier for innovation pretty high.
Apologies if this is a dumb question, but is there no "hello world"-ish sandbox for testing this theory? I can very easily imagine that trying to go head-to-head with R1 or such is going to be a boatload of GPU, but for just testing tokenizer head-to-head isn't there a smaller sized one that can be used in a bake off?