I am looking at Brenden’s GitHub repo https://github.com/bbycroft/llm-viz
Really nice stuff.
LLM Visualization
91–100 of 138 posts
Re: LLM Visualization
#92Very cool.
Re: LLM Visualization
#93The visualization I've been looking for for months. I would have happily paid serious money for this... the fact that it's free is such a gift and I don't take it for granted.
Same... this is like a textbook, but worth it
Re: LLM Visualization
#94My jaw drop to see algorhythmic complexity laid out so clearly in a 3d space like that. I wish I was smart enough to know if it's accurate or not.
To know, you must perform intellectual work, not merely be smart. I bet you are smart enough.
What a nice comment!! This has been a big failing of my mental model. I always believed if I was smart enough I should understand things without effort. Still trying to unlearn this....
Re: LLM Visualization
#95bbycroft is the GOAT!
Re: LLM Visualization
#96Expecting someone to implement an LLM in Factorio any day now, we're half-way there already with this blueprint.
Re: LLM Visualization
#97This is an excellent tool to realize how an LLM actually works from the ground up! For those reading it and going through each step, if by chance you get stuck on why 48 elements are in the first array, please refer to the model.py on minGPT [1] It's an architectural decision that it will be great to mention in the article since people without too much context might lose it [1] https://github.com/karpathy/minGPT/blob…
Are you referring specifically to line 141, which sets the number of embedding elements for gpt-nano to 48? That also seems to correspond to the Channel size C referenced in the explanation text? https://github.com/karpathy/minGPT/blob/master/mingpt/model....
That matches the name of default model selected in the right pane, "nano-gpt". I missed the "bigger picture" at first before I noticed the other models in the right pane header.
Re: LLM Visualization
#98This is an excellent tool to realize how an LLM actually works from the ground up! For those reading it and going through each step, if by chance you get stuck on why 48 elements are in the first array, please refer to the model.py on minGPT [1] It's an architectural decision that it will be great to mention in the article since people without too much context might lose it [1] https://github.com/karpathy/minGPT/blob…
Wow, I love the interactive wizzing around and the animation, very neat! Way more explanations should work like this.
I've recently finished an unorthodox kind of visualization / explanation of transformers. It's sadly not interactive, but it does have some maybe unique strengths.
First, it gives array axis semantic names, represented in the diagrams as colors (which this post also uses). So sequence axis is red, key feature dimension is green, multihead axis is orange, etc. This helps you show quite complicated array circuits and get an immediate feeling for what is going on and how different arrays are being combined with each-other. Here's a pic of the the full multihead self-attention step for example:
https://math.tali.link/raster/052n01bav6yvz_1smxhkus2qrik_07...
It also uses a kind of generalization tensor network diagrammatic notation -- if anyone remembers Penrose's tensor notation, it's like that but enriched with colors and some other ideas. Underneath these diagrams are string diagrams in a particular category, though you don't need to know (nor do I even explain that!).
Here's the main blog post introducing the formalism: https://math.tali.link/rainbow-array-algebra
Here's the section on perceptrons: https://math.tali.link/rainbow-array-algebra/#neural-network...
Here's the section on transformers: https://math.tali.link/rainbow-array-algebra/#transformers
Re: LLM Visualization
#99Am I the only one getting "Application error: a client-side exception has occurred (see the browser console for more information)." messages?
It is possible that your machine does not yet support WebGL2.
Check here : https://get.webgl.org/webgl2/
Re: LLM Visualization
#100A lot of transformer explanations fail to mention what makes self attention so powerful. Unlike traditional neural networks with fixed weights, self-attention layers adaptively weight connections between inputs based on context. This allows transformers to accomplish in a single layer what would take traditional networks multiple layers.
In case it’s confusing for anyone to see “weight” as a verb and a noun so close together, there are indeed two different things going on: 1. There are the model weights, aka the parameters. These are what get adjusted during training to do the learning part. They always exist. 2. There are attention weights. These are part of the transformer architecture and they “weight” the context of the input. They are ephemeral.…
I think in most deployments, they're not fp32 by the time you're doing inference no them, they've been quantized, possibly down to 4 bits or even fewer.
On the training side I wouldn't be surprised if they were bf16 rather than fp32.