Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding GPT tokenizers

simonwillison.net

Post reply on HN
Print view View original

41–50 of 135 posts

Re: Understanding GPT tokenizers

#41

Post by SkyPuncher »

hsjqllzlfkf wrote: post #23
SkyPuncher wrote: post #18

Earlier quoted context omitted.

Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words…

Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can you tell me anything NOT about tokenized? This is my point.

With all due respect, this feels like asking me to talk about math without talking about numbers.

Tokens are so closely tied to modern LLMs that’s it’s basically impossible to not talk about them. They’re getting a lot of attention because they are the primitive. They’re the thing of most interest for improving performance.

Re: Understanding GPT tokenizers

#42

Post by base698 »

mike_hearn wrote: post #22

A few extra notes on tokens. You don't have to use tiktoken if you aren't actually tokenizing things. The token lists are just text files that consist of the characters base64 encoded followed by the numeric ID. If you want to explore the list you can just download them and decode them yourself. I find that sorting tokens by length makes it a bit easier to get a feel for what's in there. GPT-4 has a token vocabulary…

I saw this when 3.5 came out: https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...

Haven't followed up on all the comments in it but speculates on why chain of thought improves when training on code.

Re: Understanding GPT tokenizers

#43

Post by doormatt »

thaumasiotes wrote: post #27
SkyPuncher wrote: post #18

Earlier quoted context omitted.

Tokens are the primitives that most LLMs (and broadly a lot of NLP) works with. While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. This can create some interesting challenges and unexpected behavior. It also makes certain things, like vectorization, a challenge since tokens may not map 1:1 with the words…

> While, you and I would expect whole-words to be tokens, many tokens are shorter - 3 to 4 characters - and don't always match the sentence structure you and I expect. There is a phenomenon called Broca's Aphasia which is, essentially, the inability to connect words into sentences. This mostly prevents the patient from communicating via language. But patients with this condition can reveal quite a bit about the struc…

>One of his single-word utterances, describing the mill where he works, is "Four hundred tons a day!".

I'm sorry, but I'm lost on how that's a single-word utterance.

Re: Understanding GPT tokenizers

#47

Post by RC_ITR »

throwaway2016a wrote: post #30

Pardon the n00b question, but... How does this relate to vectors? It was my understanding that the tokens were vectors and this seems to show them as an integer. It's probably a really obvious question to anyone who knows AI but I figured if I have it someone else does too.

After training, tokens are vectors, but the number of unique vectors is limited by your vocabulary size (i.e. Should 'The' get a vector or should 'Th' and 'e' each get their own vector?).

This step is deciding which clusters of letters (or whatever) get a vector and then giving them a scalar unique ID for conveniences' sake.

The training then determines what that vector actually is.

Re: Understanding GPT tokenizers

#49

Post by kenjackson »

ftxbro wrote: post #8
hsjqllzlfkf wrote: post #4

Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.

The reason it's trending today is because of the phenomenon of Glitch Tokens. They thought all Glitch Tokens had been removed by GPT-4 but apparently one is still left. If you go down the rabbit hole on Glitch Tokens it gets ... really really weird.

But does the tokenizer have anything to do with Glitch Tokens? Glitch Tokens seem more like a function of the neural network. I'm saying this with only a surface level understanding of glitch tokens.

Re: Understanding GPT tokenizers

#50

Post by CuriousSkeptic »

nighthawk454 wrote: post #35
throwaway2016a wrote: post #30

Pardon the n00b question, but... How does this relate to vectors? It was my understanding that the tokens were vectors and this seems to show them as an integer. It's probably a really obvious question to anyone who knows AI but I figured if I have it someone else does too.

The tokens are an integer. The first layer of the model is an 'embedding', which is essentially a giant lookup table. So if a string gets tokenized to Token #3, that means get the vector in row 3 of the embedding table. (Those vectors are learned during model training.) More completely, you can think of the integers as being implicitly a one-hot vector encoding. So say you have a vocab size of 20,000 and you want Tok…

Kind of refreshing to see this perspective on lookup vs matrix multiplication, specially with the bias towards the latter as more natural.

Is there some reference table somewhere mapping more code idioms like this to equivalent nn representations?

Post reply on HN