Revealing example of self-attention, the building block of transformer AI models
1–10 of 39 posts
Re: Revealing example of self-attention, the building block of transformer AI models
#2What's this about? Run this code and you'll see something?
Re: Revealing example of self-attention, the building block of transformer AI models
#3[deleted]
Re: Revealing example of self-attention, the building block of transformer AI models
#4What's this about? Run this code and you'll see something?
I think so - OP if you were to include a Jupyter notebook it would save some time
Re: Revealing example of self-attention, the building block of transformer AI models
#5Wait, is this implementation just wrong? The "num_channel" is the embed_d, the K, Q should be applied to every embedded vector, here the embedded vector has length 1 so K, Q and V should be a single scalar and being applied like torch.matmul(x_i, self.K) [sequence_dim, 1] x [1, 1], this is also intuitive because the sequence can have different size and self attention needs to work anyway.
If you want insead treat the "num_inputs" as embed_d then the attention matrix W would be 1x1 because there is only one 768 vector but instead the matrix W here is [ num_inputs, num_inputs ].
Re: Revealing example of self-attention, the building block of transformer AI models
#6What's this about? Run this code and you'll see something?
OP, just to balance out these comments, you hooked me with the readme. DNNs have been moving way faster than pedagogy can keep up and these sorts of learning experiments are super helpful.
Re: Revealing example of self-attention, the building block of transformer AI models
#7For context: The latest large language models (LLM) skip doing some complex things that previous good models did, and focus on attention. See:
"Attention is all you need" https://arxiv.org/abs/1706.03762
Re: Revealing example of self-attention, the building block of transformer AI models
#8Wait, is this implementation just wrong? The "num_channel" is the embed_d, the K, Q should be applied to every embedded vector, here the embedded vector has length 1 so K, Q and V should be a single scalar and being applied like torch.matmul(x_i, self.K) [sequence_dim, 1] x [1, 1], this is also intuitive because the sequence can have different size and self attention needs to work anyway. If you want insead treat the…
You may be right. I was not expecting this to be at the top of HN
UPDATE: I have updated the repo, attempting to address the issue raised by GaggiX
Re: Revealing example of self-attention, the building block of transformer AI models
#9Wait, is this implementation just wrong? The "num_channel" is the embed_d, the K, Q should be applied to every embedded vector, here the embedded vector has length 1 so K, Q and V should be a single scalar and being applied like torch.matmul(x_i, self.K) [sequence_dim, 1] x [1, 1], this is also intuitive because the sequence can have different size and self attention needs to work anyway. If you want insead treat the…
You may be right. I was not expecting this to be at the top of HN UPDATE: I have updated the repo, attempting to address the issue raised by GaggiX
[flagged]
Re: Revealing example of self-attention, the building block of transformer AI models
#10For context: The latest large language models (LLM) skip doing some complex things that previous good models did, and focus on attention. See: "Attention is all you need" https://arxiv.org/abs/1706.03762
This MIT lecture on recurrent neural networks provides a lot of context on why attention was so groundbreaking, and how things were solved before: https://www.youtube.com/watch?v=ySEx_Bqxvvo
And for the Attention Is All You Need paper itself, Andrej Karpathy's "GPT From Scratch" talk is a really great walkthrough of how to make this work in practice, all condensed into two hours of live coding: https://www.youtube.com/watch?v=kCc8FmEb1nY