Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding GPT tokenizers

simonwillison.net

Post reply on HN
Print view View original

21–30 of 135 posts

Re: Understanding GPT tokenizers

#21

Post by whimsicalism »

spywaregorilla wrote: post #12
ftxbro wrote: post #9

Earlier quoted context omitted.

it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.

I don't think that is intuitive at all. "Clever feature engineering" like trying to create columns from calculations of tabular data, sure. You're not going to move the needle. But the basic representation of unstructured data like text could very believably alter the need for parameters, layers, and calculation speed by orders of magnitude.

You would be wrong at the scales we are talking about.

The whole point is that it is unintuitive.

Re: Understanding GPT tokenizers

#22

Post by mike_hearn »

A few extra notes on tokens.

You don't have to use tiktoken if you aren't actually tokenizing things. The token lists are just text files that consist of the characters base64 encoded followed by the numeric ID. If you want to explore the list you can just download them and decode them yourself.

I find that sorting tokens by length makes it a bit easier to get a feel for what's in there.

GPT-4 has a token vocabulary about twice the size of GPT-3.5.

The most interesting thing to me about the GPT-4 token list is how dominated it is by non-natural languages. It's not as simple as English tokenizing more efficiently than Spanish because of frequency. The most common language after English is code. A huge number of tokens are allocated to even not very common things found in code, like "ValidateAntiForgeryToken" or "_InternalArray". From eyeballing the list I'd guess about half the tokens seem to be from source code.

My guess is that it's not a coincidence that GPT-4 both trained on a lot of code and is also the leading model. I suspect we're going to discover at some point, or maybe OpenAI already did, that training on code isn't just a neat trick to get an LLM that can knock out scripts. Maybe it's fundamentally useful to train the model to reason logically and think clearly. The highly structured and unambiguous yet also complex thought that code represents is probably a great way for the model to really level up its thought processes. Ilya Sutskever mentioned in an interview that one of the bottlenecks they face on training something smarter than GPT-4 is getting access to "more complex thought". If this is true then it's possible the Microsoft collaboration will prove an enduring competitive advantage for OpenAI, as it gives them access to the bulk GitHub corpus which is probably quite hard to scrape otherwise.

Re: Understanding GPT tokenizers

#23

Post by hsjqllzlfkf »

SkyPuncher wrote: post #18
hsjqllzlfkf wrote: post #4

Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.

Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words…

Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can you tell me anything NOT about tokenized? This is my point.

Re: Understanding GPT tokenizers

#26

Post by mike_hearn »

hsjqllzlfkf wrote: post #23
SkyPuncher wrote: post #18

Earlier quoted context omitted.

Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words…

Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can you tell me anything NOT about tokenized? This is my point.

The reason it's not discussed much is that what goes on downstream of tokenization is extremely opaque. It's lots of layers of the transformer network so the overall structure is documented but what exactly those numbers mean is hard to figure out.

There's an article here where the structure of an image generation network is explored a bit:

https://openai.com/research/sparse-transformer

They have a visualization of what the different layers are paying attention to.

There are also some good explanations of transformers elsewhere online. This one is old but I found it helpful:

http://jalammar.github.io/illustrated-transformer/

Re: Understanding GPT tokenizers

#27

Post by thaumasiotes »

SkyPuncher wrote: post #18
hsjqllzlfkf wrote: post #4

Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.

Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words…

> While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect.

There is a phenomenon called Broca's Aphasia which is, essentially, the inability to connect words into sentences. This mostly prevents the patient from communicating via language. But patients with this condition can reveal quite a bit about the structure of the language they can no longer speak.

One example discussed in The Language Instinct is someone who works at (and was injured at) a mill. He is unable to produce utterances that are more than one word long, though he seems to do well at understanding what people say to him. One of his single-word utterances, describing the mill where he works, is "Four hundred tons a day!".

This is the opposite of what you describe, a single token that is longer than one word in the base language instead of being shorter. But it appears to be the same kind of thing.

By the way, if you study a highly inflectional language such as Latin or Russian, you will lose the assumption that interpretive tokens should be whole words. You'd still expect them to align closely with sentence structure, though.

Re: Understanding GPT tokenizers

#28

Post by hsjqllzlfkf »

mike_hearn wrote: post #26
hsjqllzlfkf wrote: post #23

Earlier quoted context omitted.

Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can you tell me anything NOT about tokenized? This is my point.

The reason it's not discussed much is that what goes on downstream of tokenization is extremely opaque. It's lots of layers of the transformer network so the overall structure is documented but what exactly those numbers mean is hard to figure out. There's an article here where the structure of an image generation network is explored a bit: https://openai.com/research/sparse-transformer They have a visualization of w…

This was my suspicion, thank you.

Re: Understanding GPT tokenizers

#29

Post by api »

Has anyone ever tried a GPT trained on, say, 256 tokens representing bytes in a byte stream or even more simply binary digits?

I imagine there are efficiency trade-offs but I just wonder if it works at all.

Re: Understanding GPT tokenizers

#30

Post by throwaway2016a »

Pardon the n00b question, but...

How does this relate to vectors? It was my understanding that the tokens were vectors and this seems to show them as an integer.

It's probably a really obvious question to anyone who knows AI but I figured if I have it someone else does too.

Post reply on HN