I have been under the impression that the embedded vector is the one actually matters. Token is just another format.
Token is basically a number. Embedding is a vector consisting of those numbers (tokens).
Understanding GPT tokenizers
131–135 of 135 posts
Re: Understanding GPT tokenizers
#132Earlier quoted context omitted.
it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.
I think that’s part of the reason I would love to see somebody try it, because intuitively I think it would make a difference, but it may not. To me it’s like changing the periodic table, at the macroscopic scale it may or may not make a difference.
> I think that’s part of the reason I would love to see somebody try it, because intuitively I think it would make a difference, but it may not.
The crazy thing is it's already solved. YC should just spin a subreddit.ycombinator.com for each one, there's a nice search that works and some very nice apps for reading. What the reddit shareholders are about to buy is incredibly fragile and the management is fucking with it so much it's obvious they only care about the payola.
Re: Understanding GPT tokenizers
#133Earlier quoted context omitted.
I don't think that is intuitive at all. "Clever feature engineering" like trying to create columns from calculations of tabular data, sure. You're not going to move the needle. But the basic representation of unstructured data like text could very believably alter the need for parameters, layers, and calculation speed by orders of magnitude.
> "I don't think that is intuitive at all." That's exactly the point. Every intuition is always on the side of feature engineering.
Not really.
Re: Understanding GPT tokenizers
#134Earlier quoted context omitted.
> The token lists are just text files that consist of the characters base64 encoded followed by the numeric ID. If you want to explore the list you can just download them and decode them yourself. If you want to look at mappings for individual tokens, sure, but if you actually want to tokenize text that contains more than 1 token, the process is very non trivial. I've been writing my own JavaScript LLaMA tokenizer fo…
Would you be willing to share a GitHub link? This seems like a fun project to read through.
Published now: https://github.com/belladoreai/llama-tokenizer-js
Re: Understanding GPT tokenizers
#135A few extra notes on tokens. You don't have to use tiktoken if you aren't actually tokenizing things. The token lists are just text files that consist of the characters base64 encoded followed by the numeric ID. If you want to explore the list you can just download them and decode them yourself. I find that sorting tokens by length makes it a bit easier to get a feel for what's in there. GPT-4 has a token vocabulary…
Here is the list of the 100k GPT-4 tokens as text file. https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a0... Yes, a lot of tokens are just for code. Edit: Here as raw link for the poor mobile devices: https://gist.githubusercontent.com/s-macke/ae83f6afb89794350...
Where are all the tokens for other languages though? How are these getting tokenized?