What happened to this thread? When I saw it before it had 700+ upvotes.
LLM Visualization
81–90 of 138 posts
Re: LLM Visualization
#82The score on this post just went down by a factor of 10 and the time went to "1 hour" ago?!
Another post was merged with it: https://news.ycombinator.com/item?id=38507672
Re: LLM Visualization
#83This is an excellent tool to realize how an LLM actually works from the ground up! For those reading it and going through each step, if by chance you get stuck on why 48 elements are in the first array, please refer to the model.py on minGPT [1] It's an architectural decision that it will be great to mention in the article since people without too much context might lose it [1] https://github.com/karpathy/minGPT/blob…
Are you referring specifically to line 141, which sets the number of embedding elements for gpt-nano to 48? That also seems to correspond to the Channel size C referenced in the explanation text?
https://github.com/karpathy/minGPT/blob/master/mingpt/model....
Re: LLM Visualization
#84Earlier quoted context omitted.
I found these notes very useful. They also contain a nice summary of how LLMs/transformers work. It doesn't help that people can't seem to help taking a concept that has been around for decades (kernel smoothing) and giving it a fancy new name (attention). http://bactra.org/notebooks/nn-attention-and-transformers.ht...
It's just as bad a "convolutional neural networks" instead of "images being scaled down"
“Convolution” is a pretty well established word for taking an operation and applying it sliding-window-style across a signal. Convnets are basically just a bunch of Hough transforms with learned convolution kernels.
Re: LLM Visualization
#85This is excellent!
This is why I love Hacker News!
Re: LLM Visualization
#86amazing router
Re: LLM Visualization
#87Am I the only one getting "Application error: a client-side exception has occurred (see the browser console for more information)." messages?
Re: LLM Visualization
#88Am I the only one getting "Application error: a client-side exception has occurred (see the browser console for more information)." messages?
Same here. I blame the popularity of Next.js. More and more of the web is slowly becoming more broken on Firefox on Linux, all with the same tired error: "Application error: a client-side exception has occurred"
Re: LLM Visualization
#89Am I the only one getting "Application error: a client-side exception has occurred (see the browser console for more information)." messages?
Same here. I blame the popularity of Next.js. More and more of the web is slowly becoming more broken on Firefox on Linux, all with the same tired error: "Application error: a client-side exception has occurred"
Works fine on Firefox on Linux for me.
Re: LLM Visualization
#90This does an amazing job of showing the difference in complexity between the different models. Click on GPT-3 and you should be able to see all 4 models side-by-side. GPT-3 is a monster compared to nano-gpt.