Live data from Hacker News

From text to token: How tokenization pipelines work

Post reply on HN

Print view View original

21–22 of 22 posts

Re: From text to token: How tokenization pipelines work

#21

Post by zk0 » Sat, Dec 13, 2025, 4:20 PM UTC

tl;dr with a match statement

Re: From text to token: How tokenization pipelines work

#22

Post by anonymoushn » Tue, Dec 16, 2025, 10:29 PM UTC

i love stemming, i love searching for "anime" and getting "animal"

Post reply on HN