Worth mentioning the many other consequences of BPE tokenization: gwern.net/gpt-3#bpes https://www.lesswrong.com/posts/t9svvNPNmFf5Qa3TA/mysteries-...
Understanding GPT tokenizers
11–20 of 135 posts
Re: Understanding GPT tokenizers
#12I really really wish someone would try tokenizing off of a phonetic representation rather than textual one. I think it would be interesting to compare the output
it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.
I don't think that is intuitive at all. "Clever feature engineering" like trying to create columns from calculations of tabular data, sure. You're not going to move the needle. But the basic representation of unstructured data like text could very believably alter the need for parameters, layers, and calculation speed by orders of magnitude.
Re: Understanding GPT tokenizers
#13Earlier quoted context omitted.
it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.
I don't think that is intuitive at all. "Clever feature engineering" like trying to create columns from calculations of tabular data, sure. You're not going to move the needle. But the basic representation of unstructured data like text could very believably alter the need for parameters, layers, and calculation speed by orders of magnitude.
> "I don't think that is intuitive at all."
That's exactly the point. Every intuition is always on the side of feature engineering.
Re: Understanding GPT tokenizers
#14Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.
[deleted]
Re: Understanding GPT tokenizers
#15I really really wish someone would try tokenizing off of a phonetic representation rather than textual one. I think it would be interesting to compare the output
it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.
Most current implementations can't count syllables at all, so it would get you at least that far.
Re: Understanding GPT tokenizers
#16I really really wish someone would try tokenizing off of a phonetic representation rather than textual one. I think it would be interesting to compare the output
it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.
Yes, but how much extra layers and computing power do you need? Of course, phonetic tokens are awkward idea, but there is a reason why word "human" is encoded as only one token.
Re: Understanding GPT tokenizers
#17So the space character is part of the token?
Re: Understanding GPT tokenizers
#18Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.
Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect.
This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words you intend to weight them against.
Re: Understanding GPT tokenizers
#19So the space character is part of the token?
Yup. Most common words have several tokens - the word, the word with a capital letter, the word with a leading space and sometimes the word all in caps too.
Try searching for different words using the search box here: https://observablehq.com/@simonw/gpt-tokenizer#cell-135
Re: Understanding GPT tokenizers
#20Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.
Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words…
You are using the word vectorization in an idiosyncratic way, you are referring to the process of embedding words?