That example with the "was" token at the end of a murder novel is genius (at 3:58 - 4:28 in the video) really easy for a non technical person to understand.
Visualizing Attention, a Transformer's Heart [video]
11–20 of 181 posts
Re: Visualizing Attention, a Transformer's Heart [video]
#12Fun video. Much of my "art" lately has been dissecting models, injecting or altering attention, and creating animated visualizations of their inner workings. Some really fun shit.
Re: Visualizing Attention, a Transformer's Heart [video]
#13This video (with a slightly different title on YouTube) helped me realize that the attention mechanism isn't exactly a specific function so much as it is a meta-function. If I understand it correctly, Attention + learned weights effectively enables a Transformer to learn a semi-arbitrary function, one which involves a matching mechanism (i.e., the scaled dot-product.)
Indeed. The power of attention is that it searches the space of functions and surfaces the best function given the constraints. This is why I think linear attention will never come close to the ability of standard attention, the quadratic term is a necessary feature of searching over all pairs of inputs and outputs.
Re: Visualizing Attention, a Transformer's Heart [video]
#14That example with the "was" token at the end of a murder novel is genius (at 3:58 - 4:28 in the video) really easy for a non technical person to understand.
I think Ilya gets credit for that example — I’ve heard him use it in his interview with Jensen Huang.
Re: Visualizing Attention, a Transformer's Heart [video]
#15Fun video. Much of my "art" lately has been dissecting models, injecting or altering attention, and creating animated visualizations of their inner workings. Some really fun shit.
[deleted]
Re: Visualizing Attention, a Transformer's Heart [video]
#16Fun video. Much of my "art" lately has been dissecting models, injecting or altering attention, and creating animated visualizations of their inner workings. Some really fun shit.
Link? Sounds fun and reminds me of this tweet [0]
Re: Visualizing Attention, a Transformer's Heart [video]
#17Fun video. Much of my "art" lately has been dissecting models, injecting or altering attention, and creating animated visualizations of their inner workings. Some really fun shit.
Link? Sounds fun and reminds me of this tweet [0] [0] https://x.com/jaschasd/status/1756930242965606582
Nah someone down voted it. And yes, it looks like that + 20 others that are animated.
Re: Visualizing Attention, a Transformer's Heart [video]
#18I finally understand this! Why did every other video make it so confusing!
I'm someone who would love to get better at making educational videos/content. 3b1b is obviously the gold standard here.
I'm curious what things other videos did worse compared to 3b1b?
Re: Visualizing Attention, a Transformer's Heart [video]
#19I like the way he uses a low-rank decomposition of the Value matrix instead of Value+Output matrices. Much more intuitive!
Re: Visualizing Attention, a Transformer's Heart [video]
#20I finally understand this! Why did every other video make it so confusing!
I'm someone who would love to get better at making educational videos/content. 3b1b is obviously the gold standard here. I'm curious what things other videos did worse compared to 3b1b?
I think he had a good, intuitive understanding that he wanted to communicate and he made it come through.
I like how he was able to avoid going into the weeds and stay focused on leading you to understanding. I remember another video where I got really hung up on positional encoding and I felt like I could t continue until I understood that. Or other videos that overfocus on matrix operations or softmax, etc.