Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding GPT tokenizers

simonwillison.net

Post reply on HN
Print view View original

51–60 of 135 posts

Re: Understanding GPT tokenizers

#51

Post by sebzim4500 »

jiggawatts wrote: post #39
simonw wrote: post #19

Earlier quoted context omitted.

Yup. Most common words have several tokens - the word, the word with a capital letter, the word with a leading space and sometimes the word all in caps too. Try searching for different words using the search box here: https://observablehq.com/@simonw/gpt-tokenizer#cell-135

I wonder if the embeddings could be explicitly configured to account for these “symmetries”. E.g.: instead of storing seperate full copies of the “variants”, maybe keep a reduced representation with a common prefix and only a small subset of the embedding vector that is allowed to be learned? This could force the model to correctly learn how to capitalise, make all-caps, etc…

There was some discussion of doing this for RKVW, but I don't think it has actually been implemented yet.

The goal is simply to speed up training slightly, it wouldn't actually make a difference to the final performance of a model as big as GPT-4 (except maybe decrease the prevalence of glitch tokens)

Re: Understanding GPT tokenizers

#52

Post by sebzim4500 »

api wrote: post #29

Has anyone ever tried a GPT trained on, say, 256 tokens representing bytes in a byte stream or even more simply binary digits? I imagine there are efficiency trade-offs but I just wonder if it works at all.

I'm sure it would work, but there are obvious downsides (slower and less history) with few upsides (simpler, no glitch tokens)

Re: Understanding GPT tokenizers

#53

Post by anotherpaulg »

I frequently see people surprised about the kinds of "simple" mistakes LLMs make when given tasks that involve letters, syllables, word lengths, rhyming, etc. They don't do well at character oriented tasks which seem trivial to us, like "write me a sentence with only 5 letter words".

All of these problems stem from the fact that LLMs don't "see" the actual letters/characters in the text they are consuming and producing. They are only dealing in tokens, each of which usually blends together multiple letters.

The fact that they can sometimes or partially succeed at character-oriented tasks is the actual surprise. They are presumably using meta-knowledge about specific words. For example, they may have learned by rote that "cat" has 3 letters. Or they just know as a fact that "moon" and "tune" rhyme, without having any sense of what it means to pronounce them.

Re: Understanding GPT tokenizers

#54

Post by weinzierl »

mike_hearn wrote: post #22

A few extra notes on tokens. You don't have to use tiktoken if you aren't actually tokenizing things. The token lists are just text files that consist of the characters base64 encoded followed by the numeric ID. If you want to explore the list you can just download them and decode them yourself. I find that sorting tokens by length makes it a bit easier to get a feel for what's in there. GPT-4 has a token vocabulary…

If anyone else is looking for the list parent is mentioning I assume it is: https://openaipublic.blob.core.windows.net/encodings/r50k_ba...

Re: Understanding GPT tokenizers

#55

Post by famouswaffles »

mike_hearn wrote: post #22

A few extra notes on tokens. You don't have to use tiktoken if you aren't actually tokenizing things. The token lists are just text files that consist of the characters base64 encoded followed by the numeric ID. If you want to explore the list you can just download them and decode them yourself. I find that sorting tokens by length makes it a bit easier to get a feel for what's in there. GPT-4 has a token vocabulary…

>I suspect we're going to discover at some point, or maybe OpenAI already did, that training on code isn't just a neat trick to get an LLM that can knock out scripts.

This is a thing that's already fairly well known

https://arxiv.org/abs/2210.07128

Re: Understanding GPT tokenizers

#56

Post by jiggawatts »

sebzim4500 wrote: post #51
jiggawatts wrote: post #39

Earlier quoted context omitted.

I wonder if the embeddings could be explicitly configured to account for these “symmetries”. E.g.: instead of storing seperate full copies of the “variants”, maybe keep a reduced representation with a common prefix and only a small subset of the embedding vector that is allowed to be learned? This could force the model to correctly learn how to capitalise, make all-caps, etc…

There was some discussion of doing this for RKVW, but I don't think it has actually been implemented yet. The goal is simply to speed up training slightly, it wouldn't actually make a difference to the final performance of a model as big as GPT-4 (except maybe decrease the prevalence of glitch tokens)

> wouldn't actually make a difference to the final performance

Doesn't that assume that the embeddings learned are in some sense "perfect"? Is that actually the case in practice?

I would expect the learned embeddings to have some errors, especially for the rarer ones that have few examples available for the model to learn from.

I also thought that explicitly accounting for symmetries always improved model performance, because then it doesn't waste parameters learning things that aren't unique and interesting pieces of information.

Re: Understanding GPT tokenizers

#58

Post by jcims »

ftxbro wrote: post #9
jcims wrote: post #7

I really really wish someone would try tokenizing off of a phonetic representation rather than textual one. I think it would be interesting to compare the output

it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.

I think that’s part of the reason I would love to see somebody try it, because intuitively I think it would make a difference, but it may not.

To me it’s like changing the periodic table, at the macroscopic scale it may or may not make a difference.

Re: Understanding GPT tokenizers

#59

Post by PeterisP »

bluepoint wrote: post #17

So the space character is part of the token?

You have to represent spaces in some way (you want to make a distinction between therapist and the rapist), different tokenizers do it differently - one option is to include space as part of the token, another commonly used option is to include the lack of space as part of the token by adding a specific mark representing "the word goes on" at the end.

Re: Understanding GPT tokenizers

#60

Post by PeterisP »

jiggawatts wrote: post #56
sebzim4500 wrote: post #51

Earlier quoted context omitted.

There was some discussion of doing this for RKVW, but I don't think it has actually been implemented yet. The goal is simply to speed up training slightly, it wouldn't actually make a difference to the final performance of a model as big as GPT-4 (except maybe decrease the prevalence of glitch tokens)

> wouldn't actually make a difference to the final performance Doesn't that assume that the embeddings learned are in some sense "perfect"? Is that actually the case in practice? I would expect the learned embeddings to have some errors, especially for the rarer ones that have few examples available for the model to learn from. I also thought that explicitly accounting for symmetries always improved model performance…

Thing is, when you consider the tasks you actually want to optimize the models for, quite a few things mentioned in this discussion - e.g. correctly learn how to capitalise, make all-caps, count syllables, act on specific counts of letters - fall in the category of uninteresting things you don't want to waste parameters on. Sure, they'd help with some trick questions that refer to the peculiarities of how exactly we encode stuff in letters, but that's the whole thing we want to abstract away, going beyond textual encoding (or verbal encoding or pictures as rectangles of pixels) towards what the utterance means - like, not only we want to abstract away from spelling mistakes or variations, but also much larger changes to text like different grammar structures to say the same thing, or even saying the same thing in a different language in a different alphabet.
Post reply on HN