Does this work by just training once with next token prediction? Want to understand better how it creates fluent sentences if anyone can provide insights.
Show HN: I built a tiny LLM to demystify how language models work
131–140 of 174 posts
Re: Show HN: I built a tiny LLM to demystify how language models work
#132[dead]
Re: Show HN: I built a tiny LLM to demystify how language models work
#133[dead]
Re: Show HN: I built a tiny LLM to demystify how language models work
#134Is there some documentation for this? The code is probably the simplest (Not So) Large Language Model implementation possible, but it is not straight forward to understand for developers not familiar with multi-head attention, ReLU FFN, LayerNorm and learned positional embeddings. This projects shares similarities with Minix. Minix is still used at universities as an educational tool for teaching operating system des…
give the code to an LLM and have a discussion about it.
does this work? there is no more need for writing high level docs?
Re: Show HN: I built a tiny LLM to demystify how language models work
#135This is amazing work. Thank you.
Re: Show HN: I built a tiny LLM to demystify how language models work
#136[dead]
Re: Show HN: I built a tiny LLM to demystify how language models work
#137Earlier quoted context omitted.
give the code to an LLM and have a discussion about it.
does this work? there is no more need for writing high level docs?
LLMs can tell you what the code does but not why the developer chose to do it that way.
Also, large codebases are harder to understand. But projects like these are simple to discuss with an LLM.
Re: Show HN: I built a tiny LLM to demystify how language models work
#138Earlier quoted context omitted.
give the code to an LLM and have a discussion about it.
does this work? there is no more need for writing high level docs?
There are so many blogs and tutorials about this stuff in particular, I wouldn't worry about it being outside the training data distribution for modern LLMs. If you have a scarce topic in some obscure language I'd be more careful when learning from LLMs.
Re: Show HN: I built a tiny LLM to demystify how language models work
#139How does this compare to Andrej Karpathy's microgpt ( https://karpathy.github.io/2026/02/12/microgpt/ ) or minGPT ( https://github.com/karpathy/minGPT )?
Who cares how it compares, it's not a product it's a cool project
Microgpt isn’t a product either. Are you saying that differences between cool projects aren’t worth thinking and conversing about?
Re: Show HN: I built a tiny LLM to demystify how language models work
#140I am trying to find how the synthetic data was created (looking through the repo) and didn't find it. Maybe I am missing it - Would love to see the prompts and process on that aspect of the training data generation!
It's here:
https://github.com/arman-bd/guppylm/blob/main/guppylm/genera...
Uses a sort of mad-libs templatized style to generate all the permutations.