You beat me to the punch. I wrote a blog post[1] with the exact same title last week! Though, I went into a bit more detail with regard to embedding layers, so maybe my title is not accurate.
From text to token: How tokenization pipelines work
11–20 of 22 posts
Re: From text to token: How tokenization pipelines work
#12Oh it's good old tokenization vs for-LLM tokenizations like sentence piece or tiktoken. We shouldn't forget there are non-ML simple things like this one which doesn't ask you to buy more GPUs.
Re: From text to token: How tokenization pipelines work
#13Notably tokenization for traditional search. LLMs use very different tokenization with very different goals
100%, maybe we should do a follow up on other types of tokenization.
Re: From text to token: How tokenization pipelines work
#14Notably tokenization for traditional search. LLMs use very different tokenization with very different goals
It's a rather old-fashioned style of tokenization. In the 1980s this was common, I think. But, as noted in another comment, it doesn't work that well for languages with a richer morphology, or compounding. It's a very "English" approach.
Chinese, Japanese, Korean etc.. don’t work like this either.
However, even though the approach is “old fashioned” it’s still widely used for English. I’m not sure there is a universal approach that semantic search could use that would be both fast and accurate?
At the end of the day people choose a tokenizer that matches their language.
I will update the article to make all this clearer though!
Re: From text to token: How tokenization pipelines work
#15You beat me to the punch. I wrote a blog post[1] with the exact same title last week! Though, I went into a bit more detail with regard to embedding layers, so maybe my title is not accurate. 1. https://gafar.org/blog/text-to-tokens
Amazing, will have a read!
Re: From text to token: How tokenization pipelines work
#16Oh it's good old tokenization vs for-LLM tokenizations like sentence piece or tiktoken. We shouldn't forget there are non-ML simple things like this one which doesn't ask you to buy more GPUs.
Haha, I like “good old tokenization”
Re: From text to token: How tokenization pipelines work
#17I'm wondering if the english stopwords are not children of a forgotten declination that was forgotten from the language - ... ok so I had to check this out but I don't really have time to check more than with gemini - apparently -
The word "the" is basically the sole survivor of a massive, complex table of declensions.
In Old English, you could not just say "the." You had to choose the correct word based on gender, case, and number—exactly like you do in Polish today with ten, ta, to, tego, temu, tej, etc.
The Old English "The" (Definite Article) Case Masculine (Ten) Neuter (To) Feminine (Ta) Plural (Te) Nominative Se Þæt Sēo Þā Accusative Þone Þæt Þā Þā Genitive Þæs Þæs Þære Þāra Dative Þæm Þæm Þære Þæm Instrumental Þy Þy — —
I have read somewhere that polish was actually more precise language to be used with AI - I'm wondering if the idea of shortening words that apparently make no sense are not actually hurting it more - as noticed by the article though.
So I'm to wonder at this point - wouldn't it be worthy of exploring a tenser version of the language that might bridge that gap ? completely exploratory though I don't even know if that might be helpful idea other than being a toy
Re: From text to token: How tokenization pipelines work
#18Just curious - if we remove stop words from prompts before going to LLM, wouldn't it reduce token size? Will it keep the response from LLM same (original vs without stop tokens)?
Don't know, but GPT-5 Thinking strips out a lot of words in its reasoning trace in order to save tokens. Someone on Twitter jailbroke it in order to get the original CoT traces.
Re: From text to token: How tokenization pipelines work
#19Notably tokenization for traditional search. LLMs use very different tokenization with very different goals
It's a rather old-fashioned style of tokenization. In the 1980s this was common, I think. But, as noted in another comment, it doesn't work that well for languages with a richer morphology, or compounding. It's a very "English" approach.
This was common even in 2015. You can still see people removing stop words from text, even when they feed it to LLMs. It's of course terrible for performance, but old habits die hard I guess.
Re: From text to token: How tokenization pipelines work
#20Good explanation on tokenizing English text for regular search. But it is far from universal, and will not work well in Finnish, for example. Folding diacritics makes "vähä" (little) into "vaha" (wax). Dropping stop words like "The" misses the word for "tea" (in rather old-fashioned finnish, but also in current Danish). Stemming Finnish words is also much more complex, as we tend to append suffixes to the words inste…
Yep and I find that this really worsens LLM performance. For example `Ben,Alice` would be tokenized as `Ben|,A|lice`. And having to connect `lice` to the name `Alice` does not make it any easier for LLMs. However, formatting it as `Ben, Alice` tokenizes it as `Ben|,| Alice`. I found it kind of useful to improve performance by just formatting the data a bit differently.
I actually just started working on a data formatter that applies principles like these to drastically reduce the amount of tokens without decreasing the performance, like other formats do (looking at you, tson).