TokenTown: A visual way to understand how LLMs work
laurentiugabriel.github.io
TokenTown: A visual way to understand how LLMs work
1–10 of 25 posts
Re: TokenTown: A visual way to understand how LLMs work
#2You can paste text and immediately see how it's split into tokens, along with token IDs, counts, and other details. It's useful for understanding why prompts cost what they do, why context limits behave the way they do, and why models sometimes split words in unexpected places.
I made it because I kept explaining tokenization to friends and realized there wasn't a simple, interactive tool that focused on learning rather than documentation.
I'd love feedback on:
Features that would make it more useful UI improvements Bugs or edge cases you've found
Hope it's useful!
Re: TokenTown: A visual way to understand how LLMs work
#3this doesnt help at all
Re: TokenTown: A visual way to understand how LLMs work
#4I think you need to know a lot to understand this, so perhaps it should mention what you should already know in order for it to be comprehensible.
Re: TokenTown: A visual way to understand how LLMs work
#5this doesnt help at all
Congratulations you're now a senior AI engineer
Re: TokenTown: A visual way to understand how LLMs work
#6This would be really cool if it did 3Brown 1Blue's style of visualization - but interactively, and allowed you to explore at any stage.
Re: TokenTown: A visual way to understand how LLMs work
#7I read through the explanations and they’re all typical hand wavy LLM prose, full of jargon, and random tangential details. If you’re going to publicly present this as a learning tool, it would be great if you spent some actual human brain cycles writing and polishing it by hand, rather than just chucking the first thing that claude spits out over the fence. I can pretty much guess the prompt you used and probably get the same thing.
Re: TokenTown: A visual way to understand how LLMs work
#8this doesnt help at all
Just softmax it over the KV cache dude!
Re: TokenTown: A visual way to understand how LLMs work
#9You can paste text and immediately see how it's split into tokens, along with token IDs, counts, and other details. It's useful for understanding why prompts cost what they do, why context limits behave the way they do, and why models sometimes split words in unexpected places. I made it because I kept explaining tokenization to friends and realized there wasn't a simple, interactive tool that focused on learning rat…
> I made it
*I asked a LLM to make it
Re: TokenTown: A visual way to understand how LLMs work
#10Can we filter out random vibe-coded projects like this from the homepage?