Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding GPT tokenizers

simonwillison.net

Post reply on HN
Print view View original

121–130 of 135 posts

Re: Understanding GPT tokenizers

#121

Post by belladoreai »

mike_hearn wrote: post #119
belladoreai wrote: post #108

Earlier quoted context omitted.

TikToken doesn't provide a tokenizer that's compatible with LLaMA.

Ah interesting. What's the difference? Isn't it just finding the minimal mapping of character sequences to numbers?

That's what I thought when I started working on this, but it turns out the answer is no! This approach - "minimal mapping of character sequences to numbers" - can be described as a greedy algorithm. Using this approach will often produce correct results, but not always. I can provide an example:

Input string: " grabbed"

Tokenize that with the greedy algorithm, you get [17229, 2580] == [" grab", "bed"]

Tokenize that with actual LLaMA tokenizer, you get [2646, 1327, 287] == [" gra", "bb", "ed"]

Note that the correct tokenizer represents this string with 3 tokens, even though it would be more efficient to represent this string with 2 tokens (yes, those 2 tokens exist in the vocabulary).

LLaMA uses SentencePiece Byte-Pair Encoding for tokenization, and it has many weird quirks like this.

Re: Understanding GPT tokenizers

#122

Post by gwern »

zarzavat wrote: post #78
gwern wrote: post #67

Earlier quoted context omitted.

> However, once they've started writing rhyming poetry it's hard to get them to stop rhyming. They seem to have formed a strong association between rhyming and poetry. I've also been unable to get them to obey a specific rhyming scheme like ABBAB. Correct and commonly observed (eg. https://arxiv.org/abs/2305.11064 ). (At least, for GPT models. I don't know as much about the Anthropic models as I should, although I un…

I wonder if having access to characters actually helps rhyming in English all that much, as English rules of pronunciation are essentially rote-learned anyway. If it were not rote-learning, then it might make different mistakes, for example expecting two words to rhyme because they end with the same suffix. Perhaps it would be more effective to ask it to produce poems in the format: English0 IPA0 English1 IPA1 , wher…

I did some poking around with IPA way back in 2020 reasoning that if the phonetics were explicit maybe that'd be fine, but I didn't get anything that looked like a big improvement: https://gwern.net/gpt-3#ipa-rhyme-annotations My guess was that it doesn't see enough IPA to use it that way zero-shot, and the erasure of information by BPEs damages its learning fundamentally enough that you can't easily prompt your way into anything better.

I speculated that because it's memorized so much, it shouldn't be too hard for it to learn to rhyme properly if you finetuned it on an IPA-encoded or non-BPE-tokenized poetry corpus, but I never got around to it and AFAIK no one else has tried that yet.

Re: Understanding GPT tokenizers

#123

Post by nighthawk454 »

quickthrower2 wrote: post #89
CuriousSkeptic wrote: post #50

Earlier quoted context omitted.

Kind of refreshing to see this perspective on lookup vs matrix multiplication, specially with the bias towards the latter as more natural. Is there some reference table somewhere mapping more code idioms like this to equivalent nn representations?

Andrej covers this in https://github.com/karpathy/nn-zero-to-hero . He explains things in multiple ways, both the matrix multiplications as well as the "programmer's" way of thinking of it - i.e. the lookups. The downside is it takes a while to get through those lectures. I would say for each 1 hour you need another 10 to looks stuff up and practice, unless you are fresh out of calculus and linear algebra classes. Ot…

Thanks! I'm gonna check that out

Re: Understanding GPT tokenizers

#124

Post by mike_hearn »

resters wrote: post #113
mike_hearn wrote: post #22

A few extra notes on tokens. You don't have to use tiktoken if you aren't actually tokenizing things. The token lists are just text files that consist of the characters base64 encoded followed by the numeric ID. If you want to explore the list you can just download them and decode them yourself. I find that sorting tokens by length makes it a bit easier to get a feel for what's in there. GPT-4 has a token vocabulary…

One thing I find fascinating about GPT-4 (and I'm curious about your take) is that it can not only generate novel, non-trivial code, but it can (upon request) output that code as a base64 encoded string... seemingly all from the model itself.

I don't have a great explanation for that, other than the obvious trivial one that it must have seen a lot of base64 encoded text alongside the decoded text in its training set, and that was sufficient for a small part of the network to learn how to decode it. If you look at visualizations of smaller RNNs trained on code then you can identify neurons that activate for things like "inside a quoted string", "inside the expression of an if statement" and so on.

Re: Understanding GPT tokenizers

#125

Post by washadjeffmad »

wolfgang42 wrote: post #76
hsjqllzlfkf wrote: post #23

Earlier quoted context omitted.

Your answer explains what tokenizers are, which isn't what I asked. You also told me something interesting about tokenizers, which is also not what I asked. Can you tell me anything NOT about tokenized? This is my point.

You asked why there was a focus on discussing tokenizers, and got an answer explaining why tokenizers are something people would want to discuss.

> ...It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.

If someone points out a preponderance of information on one step relative to all other steps, they probably are not asking for even more information about that step.

People like to chip in with what they've recently learned, so one answer is that most people on HN don't understand much beyond the input layer. A better answer is that the relative complexity of the processes in subsequent layers increases substantially, along with the requisite background to understand them. They also don't share the relative commonality of the input layer, so fewer people are qualified to discuss them with any authority.

That's where I am, so I get it. I'm working on building learning resources for a symposium, and it feels very much like "Step 1: Tokenize, Step 2: ???, Step 3: Output!".

Re: Understanding GPT tokenizers

#126

Post by washadjeffmad »

plewd wrote: post #64
samstave wrote: post #62

I didnt fully understand tokens, and I went down this fun rabbit hole with GPT: https://chat.openai.com/share/b8f06d5e-f2d9-47d7-9c60-69b088... - it turned into me asking it to help me with an "understanding AI" book definition, I learned a LOT in that thread.

Asking ChatGPT to develop a learning path/syllabus to learn a topic seems really effective, I've never thought to try that before.

One of the first things I did after I found a reasonably performant local LLM was create a syllabus and learning objectives for each of the topics, then use it to develop some problem sets.

They're very much GIGO without LoRA, so you need the concepts and vocabulary to direct its output. Try it with a subject you have a lot of domain knowledge in; basic questions won't give you complete answers. A lot of output is completely a function of your prompting.

Re: Understanding GPT tokenizers

#127

Post by Ambix »

kgeist wrote: post #74
simonw wrote: post #72

Earlier quoted context omitted.

My current understanding is that the lack of a token for a specific word does nothing to prevent that word from being "understood" or produced in output - GPT-4 is very capable of consuming and producing text in languages such as Spanish despite most Spanish words not corresponding to a single token.

For Russian text, it degrades to, basically, 1 character = 1 token, due to the tokenization issues discussed in the article, yet it produces absolutely coherent text, almost same as in English. In my tests, its Russian output is worse than English output, though, something like 80% quality I'd say. I'm not an LLM expert but I have a theory that, being mostly trained on English text, its thought processes actually hap…

I think the same. LLMs are actually sort of "multi-linguas", able to transform source of any language to internal representation and then do output in some other language, thanks to so many layers of neurons inside it.

Re: Understanding GPT tokenizers

#128

Post by Ambix »

For those who interested, there some new researches in the field [0]. It usually possible to create more compact token representation from given text, but my guess the greedy "optimal" tokenizer might harm the performance of the model?

[0] https://www.reddit.com/r/LocalLLaMA/comments/140gcn7/new_tok...

Re: Understanding GPT tokenizers

#129

Post by Ambix »

szopa wrote: post #86

How I wish this post had appeared a few days earlier... I am writing on my own library for some agent experiments (in go, to make my life more interesting I guess), and knowing the number of tokens is important to implement a token buffer memory (as you approach the model's context window size, you prune enough messages from the beginning of the conversation that the whole thing keeps some given size, in tokens). Whi…

You might reuse simple LLaMA tokenizer right in your Go code, look there:

https://github.com/gotzmann/llama.go/blob/8cc54ca81e6bfbce25...

Re: Understanding GPT tokenizers

#130

Post by Ambix »

throwaway2016a wrote: post #30

Pardon the n00b question, but... How does this relate to vectors? It was my understanding that the tokens were vectors and this seems to show them as an integer. It's probably a really obvious question to anyone who knows AI but I figured if I have it someone else does too.

Tokens are just integer numbers, showing their position in the big vocabulary - it's that simple :)

And vocabulary is just an array / vector / list - it depends which programming language you use, each has each own terminology for that data structure.

For example LLaMA vocabulary has 32,000 tokens.

Post reply on HN