Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding GPT tokenizers

simonwillison.net

Post reply on HN
Print view View original

1–10 of 135 posts

Re: Understanding GPT tokenizers

#4

Post by hsjqllzlfkf »

Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.

Re: Understanding GPT tokenizers

#6

Post by simonw »

hsjqllzlfkf wrote: post #4

Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.

I just think they're interesting.

From a practical point of view they only really matter in that we have to think carefully about how to use our token budget.

Re: Understanding GPT tokenizers

#8

Post by ftxbro »

hsjqllzlfkf wrote: post #4

Could anyone who's an expert comment why there seems to be such a focus on discussing tokenizers? It seems every other day there's a new article or implementation of a tokenizer on HN. But downstream from that, rarely anything. As a non-expert I would have thought to tokenizing is just one step.

The reason it's trending today is because of the phenomenon of Glitch Tokens. They thought all Glitch Tokens had been removed by GPT-4 but apparently one is still left. If you go down the rabbit hole on Glitch Tokens it gets ... really really weird.

Re: Understanding GPT tokenizers

#9

Post by ftxbro »

jcims wrote: post #7

I really really wish someone would try tokenizing off of a phonetic representation rather than textual one. I think it would be interesting to compare the output

it doesn't matter, the 'bitter lesson' as coined by Rich Sutton is that stacking more layers with more parameters and compute and dataset size is going to swamp any kind of clever 'feature engineering' like trying to be clever about phonetic tokens. Karpathy for example just wants to go back to byte tokens.
Post reply on HN