Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding GPT tokenizers

simonwillison.net

Post reply on HN
Print view View original

81–90 of 135 posts

Re: Understanding GPT tokenizers

#81

Post by Der_Einzige »

jcims wrote: post #7

I really really wish someone would try tokenizing off of a phonetic representation rather than textual one. I think it would be interesting to compare the output

I agree with you, and I'm SHOCKED at how little work there actually is in phonetics within the NLP community. Consider that most of the phonetic tools that I am using to enforce rhyming or similar syntactic constrained in constrained text generation studio (https://github.com/Hellisotherpeople/Constrained-Text-Genera...) were built circa 2014, such as the CMU rhyming dictionary. In most cases, I could not find better modern implementations of these tools.

I did learn an awful lot about phonetic representations and matching algorithms. Things like "soundex" and "double metaphone" now make sense to me and are fascinating to read about.

Re: Understanding GPT tokenizers

#82

Post by Der_Einzige »

gwern wrote: post #11

Worth mentioning the many other consequences of BPE tokenization: gwern.net/gpt-3#bpes https://www.lesswrong.com/posts/t9svvNPNmFf5Qa3TA/mysteries-...

My paper, titled "Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio", is cited in that gwern article!

https://paperswithcode.com/paper/most-language-models-can-be...

Re: Understanding GPT tokenizers

#83

Post by Der_Einzige »

hsjqllzlfkf wrote: post #4

Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.

Most of the shitty behavior of LLMs on syntactic and lexical tasks are due to the tokenizer and not due to the LLM itself. Having even tiny changes in tokenization has massive downstream effects on LLM behavior.

Re: Understanding GPT tokenizers

#84

Post by Der_Einzige »

ftxbro wrote: post #9
jcims wrote: post #7

I really really wish someone would try tokenizing off of a phonetic representation rather than textual one. I think it would be interesting to compare the output

it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.

If we don't fix up issues caused by the tokenizers, than techniques which literally remove superfluous computation (i.e. through filters of the LLM probability distribution) are useful as a stop-gap.

Switching to bytes is the ultimate fix, but for the interim, if you want reliable rhyming with an LLM, you need filter-assisted decoding: https://paperswithcode.com/paper/most-language-models-can-be... and replicas post about this work: https://replicate.com/blog/turn-your-llm-into-a-poet

Re: Understanding GPT tokenizers

#85

Post by GeneralMayhem »

kenjackson wrote: post #49
ftxbro wrote: post #8

Earlier quoted context omitted.

The reason it's trending today is because of the phenomenon of Glitch Tokens. They thought all Glitch Tokens had been removed by GPT-4 but apparently one is still left. If you go down the rabbit hole on Glitch Tokens it gets ... really really weird.

But does the tokenizer have anything to do with Glitch Tokens? Glitch Tokens seem more like a function of the neural network. I'm saying this with only a surface level understanding of glitch tokens.

It does a bit, because the fact that they're able to persist is sort of an artifact of how naive the tokenizer is (it's a counting operation based on n-grams), and that it runs as a separate step. There's no feedback from the transformer to the tokenizer to say "hey, this token is actually pretty meaningless, maybe try again on that one". That means that strings of characters that are common but very low semantic value, like the example of Reddit usernames that mostly post on /r/counting, will be included in the model's vocabulary even though they're not interesting.

When humans see extremely low-information-density data, we can forget it. And the model can too, but only kind of - it can forget (or rather, never learn) what the "word" means, but it can't forget that it's a word.

Re: Understanding GPT tokenizers

#86

Post by szopa »

How I wish this post had appeared a few days earlier... I am writing on my own library for some agent experiments (in go, to make my life more interesting I guess), and knowing the number of tokens is important to implement a token buffer memory (as you approach the model's context window size, you prune enough messages from the beginning of the conversation that the whole thing keeps some given size, in tokens). While there's a nice native library in go for OpenAI models (https://github.com/tiktoken-go/tokenizer), the only library I found for Hugging Face models (and Claude, they published their tokenizer spec in the same JSON format) calls into HF's Rust implementation, which makes it challenging as a dependency in Go. What is more, any tokenizer needs to keep some representation of its vocabulary in memory. So, in the end I removed the true tokenizers, and ended up using an approximate version (just split it in on spaces and multiply by a factor I determined experimentally for the models I use using the real tokenizer, with a little extra for safety). If it turns out someone needs the real thing they can always provide their own token counter). I was actually rather happy with this result: I have less dependencies, and use less memory. But to get there I needed to do a deep dive too understand BPE tokenizers :)

(The library, if anyone is interested: https://github.com/ryszard/agency.)

Re: Understanding GPT tokenizers

#87

Post by s-macke »

mike_hearn wrote: post #22

A few extra notes on tokens. You don't have to use tiktoken if you aren't actually tokenizing things. The token lists are just text files that consist of the characters base64 encoded followed by the numeric ID. If you want to explore the list you can just download them and decode them yourself. I find that sorting tokens by length makes it a bit easier to get a feel for what's in there. GPT-4 has a token vocabulary…

Here is the list of the 100k GPT-4 tokens as text file.

https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a0...

Yes, a lot of tokens are just for code.

Edit: Here as raw link for the poor mobile devices:

https://gist.githubusercontent.com/s-macke/ae83f6afb89794350...

Re: Understanding GPT tokenizers

#88

Post by yorwba »

whimsicalism wrote: post #21
spywaregorilla wrote: post #12

Earlier quoted context omitted.

I don't think that is intuitive at all. "Clever feature engineering" like trying to create columns from calculations of tabular data, sure. You're not going to move the needle. But the basic representation of unstructured data like text could very believably alter the need for parameters, layers, and calculation speed by orders of magnitude.

You would be wrong at the scales we are talking about. The whole point is that it is unintuitive.

If you replace a tokenizer with 5 bytes per token on average by a byte-level representation, you now need 5 times as much memory and (depending on the specifics of the attention mechanism) 11 to 25 times as much compute.

At the scales we're talking about, that's quite a hefty price to pay, and it doesn't even take into account that you might need more layers to replace the processing that was implicitly done by the tokenizer.

Re: Understanding GPT tokenizers

#89

Post by quickthrower2 »

CuriousSkeptic wrote: post #50
nighthawk454 wrote: post #35

Earlier quoted context omitted.

The tokens are an integer. The first layer of the model is an 'embedding', which is essentially a giant lookup table. So if a string gets tokenized to Token #3, that means get the vector in row 3 of the embedding table. (Those vectors are learned during model training.) More completely, you can think of the integers as being implicitly a one-hot vector encoding. So say you have a vocab size of 20,000 and you want Tok…

Kind of refreshing to see this perspective on lookup vs matrix multiplication, specially with the bias towards the latter as more natural. Is there some reference table somewhere mapping more code idioms like this to equivalent nn representations?

Andrej covers this in https://github.com/karpathy/nn-zero-to-hero. He explains things in multiple ways, both the matrix multiplications as well as the "programmer's" way of thinking of it - i.e. the lookups. The downside is it takes a while to get through those lectures. I would say for each 1 hour you need another 10 to looks stuff up and practice, unless you are fresh out of calculus and linear algebra classes.

Other idioms I can think of, in my words:

Softmax = take the maximum (but in a differentiable way)

tanh/sigmoid/relu = a switch. "activation"

cross entropy loss = average(-log(probability you gave to the right answer)). Averaged over the current batch you are training on for this step. (Sorry that is still quite mathy).

Re: Understanding GPT tokenizers

#90

Post by isoprophlex »

Does anyone have any understanding why these models don't simply output unicode, chunked into 12 or 16 bit words or whatever? The token lists are 100.000 tokens long, 65336 positions derived from bit sequences shouldn't be a problem right?
Post reply on HN