11–20 of 27 posts
I'd love to understand how LLMs work, but this site assumed a bit too much knowledge for me to get much from it. Looks cool though.
Wtok and Wpos should be 26-dim along one of the axis but it shows a 16x16 matrix be default, fc1 instead 16x64 with the default settings (not 16x16).
To give a sense of what the loss value means, maybe you can add a small explainer section as a question and add this explanation from Karpathy’s blog:
> Over 1,000 steps the loss decreases from around 3.3 (random guessing among 27 tokens: −log(1/27)≈3.3) down to around 2.37.
to reiterate that the model is being trained to predict the next token out of 27 possible tokens and is now doing better than the baseline of random guess.
I was a little confused by "see, its much better" when the output is stuff like isovrak and kucey. What is it supposed to be generating?
https://karpathy.github.io/2026/02/12/microgpt/
It was submitted to hn a few days ago but only received a few comments. https://news.ycombinator.com/item?id=47000263
I'd love to understand how LLMs work, but this site assumed a bit too much knowledge for me to get much from it. Looks cool though.
About how many training steps are required to get good output?