The Illustrated Transformer
jalammar.github.io
The Illustrated Transformer
1–10 of 92 posts
Re: The Illustrated Transformer
#2Haven't watched it yet...
...but, if you have favorite resources on understanding Q & K, please drop them in comments below...
(I've watched the Grant Sanderson/3blue1brown videos [including his excellent talk at TNG Big Tech Day '24], but Q & K still escape me).
Thank you in advance.
Re: The Illustrated Transformer
#3Haven't watched it yet... ...but, if you have favorite resources on understanding Q & K, please drop them in comments below... (I've watched the Grant Sanderson/3blue1brown videos [including his excellent talk at TNG Big Tech Day '24], but Q & K still escape me). Thank you in advance.
Implement transformers yourself (ie in Numpy). You'll never truly understand it by just watching videos.
Re: The Illustrated Transformer
#4Haven't watched it yet... ...but, if you have favorite resources on understanding Q & K, please drop them in comments below... (I've watched the Grant Sanderson/3blue1brown videos [including his excellent talk at TNG Big Tech Day '24], but Q & K still escape me). Thank you in advance.
I think this video does a pretty good job explaining it, starting about 10:30 minutes in: https://www.youtube.com/watch?v=S27pHKBEp30
Re: The Illustrated Transformer
#5Haven't watched it yet... ...but, if you have favorite resources on understanding Q & K, please drop them in comments below... (I've watched the Grant Sanderson/3blue1brown videos [including his excellent talk at TNG Big Tech Day '24], but Q & K still escape me). Thank you in advance.
I think this video does a pretty good job explaining it, starting about 10:30 minutes in: https://www.youtube.com/watch?v=S27pHKBEp30
As the first comment says "This aged like fine wine". Six years old, but the fundamentals haven't changed.
Re: The Illustrated Transformer
#6Here's the comment from the author himself (jayalammar) talking about other good resources on learning Transformers:
Re: The Illustrated Transformer
#7Kudos also to Transformer Explainer team for putting some amazing visualizations https://poloclub.github.io/transformer-explainer/
It really clicked to me after reading this two and watching 3blue1brown videos
Re: The Illustrated Transformer
#8I have this book. Really a life savior to help me catching up a few months ago when my team decided to use LLMs in our systems.
Re: The Illustrated Transformer
#9Haven't watched it yet... ...but, if you have favorite resources on understanding Q & K, please drop them in comments below... (I've watched the Grant Sanderson/3blue1brown videos [including his excellent talk at TNG Big Tech Day '24], but Q & K still escape me). Thank you in advance.
Implement transformers yourself (ie in Numpy). You'll never truly understand it by just watching videos.
Seconding this, the terms "Query" and "Value" are largely arbitrary and meaningless in practice, look at how to implement this in PyTorch and you'll see these are just weight matrices that implement a projection of sorts, and self-attention is always just self_attention(x, x, x) or self_attention(x, x, y) in some cases, where x and y are are outputs from previous layers.
Plus with different forms of attention, e.g. merged attention, and the research into why / how attention mechanisms might actually be working, the whole "they are motivated by key-value stores" thing starts to look really bogus. Really it is that the attention layer allows for modeling correlations and/or multiplicative interactions among a dimension-reduced representation.
Re: The Illustrated Transformer
#10Earlier quoted context omitted.
Implement transformers yourself (ie in Numpy). You'll never truly understand it by just watching videos.
Seconding this, the terms "Query" and "Value" are largely arbitrary and meaningless in practice, look at how to implement this in PyTorch and you'll see these are just weight matrices that implement a projection of sorts, and self-attention is always just self_attention(x, x, x) or self_attention(x, x, y) in some cases, where x and y are are outputs from previous layers. Plus with different forms of attention, e.g. m…
Do you think the dimension reduction is necessary? Or is it just practical (due to current hardware scarcity)?