Visualizing Attention, a Transformer's Heart [video]
3blue1brown.com
Visualizing Attention, a Transformer's Heart [video]
1–10 of 181 posts
Re: Visualizing Attention, a Transformer's Heart [video]
#2His previous post 'But what is a GPT?' is also really good: https://www.3blue1brown.com/lessons/gpt
Re: Visualizing Attention, a Transformer's Heart [video]
#3Working in a closely related space and this instantly became part of my team's onboarding docs.
Worth noting that a lot of the visualization code is available in Github.
https://github.com/3b1b/videos/tree/master/_2024/transformer...
Re: Visualizing Attention, a Transformer's Heart [video]
#4Working in a closely related space and this instantly became part of my team's onboarding docs. Worth noting that a lot of the visualization code is available in Github. https://github.com/3b1b/videos/tree/master/_2024/transformer...
Sounds interesting; what else is part of those onboarding docs?
Re: Visualizing Attention, a Transformer's Heart [video]
#5It always blows my mind that Grant Sanderson can explain complex topics in such a clear, understandable way.
I've seen several tutorials, visualisations, and blogs explaining Transformers, but I didn't fully understand them until this video.
Re: Visualizing Attention, a Transformer's Heart [video]
#6I finally understand this! Why did every other video make it so confusing!
Re: Visualizing Attention, a Transformer's Heart [video]
#7I finally understand this! Why did every other video make it so confusing!
It is confusing, 3b1b is just that good.
Re: Visualizing Attention, a Transformer's Heart [video]
#8I finally understand this! Why did every other video make it so confusing!
Because:
1. good communication requires an intelligence that most people sadly lack
2. because the type of people who are smart enough to invent transformers have zero incentive to make them easily understandable.
most documents are written by authors subconsciously desperate to mentally flex on their peers.
Re: Visualizing Attention, a Transformer's Heart [video]
#9This video (with a slightly different title on YouTube) helped me realize that the attention mechanism isn't exactly a specific function so much as it is a meta-function. If I understand it correctly, Attention + learned weights effectively enables a Transformer to learn a semi-arbitrary function, one which involves a matching mechanism (i.e., the scaled dot-product.)
Re: Visualizing Attention, a Transformer's Heart [video]
#10It always blows my mind that Grant Sanderson can explain complex topics in such a clear, understandable way. I've seen several tutorials, visualisations, and blogs explaining Transformers, but I didn't fully understand them until this video.
His content and impact is phenomenal