Viewing profile — tokfan

tokfan

HN member

No profile information was provided.

story
Tokenizers
Sun, Aug 17, 2025, 7:40 PM UTC
story
LLM Tokenization Demo
Wed, Aug 13, 2025, 8:18 PM UTC
comment
Comment #44893312
Same, same. However I have more of Markdown file where I have different headings for different projects and then use the "check" function of Markdown.

Wed, Aug 13, 2025, 8:17 PM UTC
story
Sampling – Model Context Protocol
Tue, Aug 12, 2025, 8:47 PM UTC
story
What Is LLM Tokenization and Why Is It Important?
Mon, Aug 11, 2025, 8:57 PM UTC
story
All you need to know about Tokenization in LLMs
Sun, Aug 10, 2025, 10:07 AM UTC
comment
Comment #44834525
Seems to be paywalled? I get a weird thing when scrolling down.

Fri, Aug 08, 2025, 7:36 AM UTC
story
Tokenization in Large Language Models
Fri, Aug 08, 2025, 7:35 AM UTC
comment
Comment #44834509
Haha, this is hilarious. Its interesting to see how even with "Phd level skills" LLMs often fail to answer such basic questions

Fri, Aug 08, 2025, 7:34 AM UTC