Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Revealing example of self-attention, the building block of transformer AI models

github.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: Revealing example of self-attention, the building block of transformer AI models

#21

Post by jostmey »

GaggiX wrote: post #17
jostmey wrote: post #8

Earlier quoted context omitted.

You may be right. I was not expecting this to be at the top of HN UPDATE: I have updated the repo, attempting to address the issue raised by GaggiX

Now that you have updated the code I think you have just flipped the error, it's still there, just doing the opposite (still wrong) thing: for example, if "num_steps" is 768 and "num_inputs" is 1 then K, Q, V is 1x1, okay but the the attention matrix cannot also be 1x1, it needs to be 768x768; you can also do the opposite K, Q, V being 768x768 and the attention matrix 1x1, before the error was that you made both the…

I just checked and the attention matrix is 768x768. Basically, I reshape each sample to be a single step with an embed size of 768. I hope this makes sense. I may rewrite the README later tonight to make this point clear.

Let me know if you think I still don't got it!!!

Re: Revealing example of self-attention, the building block of transformer AI models

#22

Post by GaggiX »

jostmey wrote: post #21
GaggiX wrote: post #17

Earlier quoted context omitted.

Now that you have updated the code I think you have just flipped the error, it's still there, just doing the opposite (still wrong) thing: for example, if "num_steps" is 768 and "num_inputs" is 1 then K, Q, V is 1x1, okay but the the attention matrix cannot also be 1x1, it needs to be 768x768; you can also do the opposite K, Q, V being 768x768 and the attention matrix 1x1, before the error was that you made both the…

I just checked and the attention matrix is 768x768. Basically, I reshape each sample to be a single step with an embed size of 768. I hope this makes sense. I may rewrite the README later tonight to make this point clear. Let me know if you think I still don't got it!!!

If you have a single embedded vector, it should be clear that the self-attention matrix cannot be an arbitrary NxN but only a 1x1, follow the fix I have added to my previous comment.

Also if K, Q, V and the self-attention matrix have the same dimensions even when the embed_d and sequence dim have different sizes there is something wrong.

Re: Revealing example of self-attention, the building block of transformer AI models

#23

Post by ntonozzi »

cshimmin wrote: post #19

TBH I don't find this exposition enlightening. If you don't already know the pytorch API, the code will be pretty obscure. And if you do understand the API, it's still not very clarifying. Instead, it's a lot easier IMO to understand self attention in just plain English. If you understand the words, and also understand the torch API (or tensorflow or numpy, they're all more or less fungible), then it should also be s…

Thanks for the helpful explanation. What is the context length in this explanation?

Re: Revealing example of self-attention, the building block of transformer AI models

#25

Post by two_in_one »

ntonozzi wrote: post #23
cshimmin wrote: post #19

TBH I don't find this exposition enlightening. If you don't already know the pytorch API, the code will be pretty obscure. And if you do understand the API, it's still not very clarifying. Instead, it's a lot easier IMO to understand self attention in just plain English. If you understand the words, and also understand the torch API (or tensorflow or numpy, they're all more or less fungible), then it should also be s…

Thanks for the helpful explanation. What is the context length in this explanation?

my guess it's N here

Re: Revealing example of self-attention, the building block of transformer AI models

#26

Post by two_in_one »

cshimmin wrote: post #19

TBH I don't find this exposition enlightening. If you don't already know the pytorch API, the code will be pretty obscure. And if you do understand the API, it's still not very clarifying. Instead, it's a lot easier IMO to understand self attention in just plain English. If you understand the words, and also understand the torch API (or tensorflow or numpy, they're all more or less fungible), then it should also be s…

Correct me if I'm wrong. Here Q, K, and V are trainable. d_k and d_v are selectable metaparameters(?) d_f - input vectors' size, N is their number, or window's size. Next time we usually process another non-overlapping set of input vectors.

Re: Revealing example of self-attention, the building block of transformer AI models

#27

Post by fireworkcrayon »

cshimmin wrote: post #19

TBH I don't find this exposition enlightening. If you don't already know the pytorch API, the code will be pretty obscure. And if you do understand the API, it's still not very clarifying. Instead, it's a lot easier IMO to understand self attention in just plain English. If you understand the words, and also understand the torch API (or tensorflow or numpy, they're all more or less fungible), then it should also be s…

> You have a collection of N things, with some fixed number of features each (doesn't matter how many, but call it d_f). Call that collection {x_i}. You have two learnable matrices Q and K, which learn to project those items into some collection of "questions" and "answers", respectively. Doesn't really matter what those questions or answers are, the NN will figure out what they should be during training. These projection matrices map x_i -> k_i, and x_i -> q_i. k and q are each d_k-dimensional vectors (d_k is a number you choose) of features, so Q and K are d_k-by-d_f matrices.

Wait: this is the plain English version? I’ve just been presented an alphabet soup of symbols, most of which haven’t been defined:

N, d, f, x, i, Q, K, and q and k (lowercase). The only one I’m sure I understand is N.

The different letters seem to be drawn from certain regions of the alphabet, so I guess that’s significant in some way?

(I do understand vector, matrix, softmax, dimensionality, etc.)

I’m embarrassed to admit this here, but I run into this issue almost any time concepts are explained in math, including in the textbooks that teach math. What am I missing?

Re: Revealing example of self-attention, the building block of transformer AI models

#28

Post by jostmey »

GaggiX wrote: post #22
jostmey wrote: post #21

Earlier quoted context omitted.

I just checked and the attention matrix is 768x768. Basically, I reshape each sample to be a single step with an embed size of 768. I hope this makes sense. I may rewrite the README later tonight to make this point clear. Let me know if you think I still don't got it!!!

If you have a single embedded vector, it should be clear that the self-attention matrix cannot be an arbitrary NxN but only a 1x1, follow the fix I have added to my previous comment. Also if K, Q, V and the self-attention matrix have the same dimensions even when the embed_d and sequence dim have different sizes there is something wrong.

Okay, I think it is fixed. I appreciate your help. If you see this, let me know if the issue still exists!

Re: Revealing example of self-attention, the building block of transformer AI models

#29

Post by cshimmin »

fireworkcrayon wrote: post #27
cshimmin wrote: post #19

TBH I don't find this exposition enlightening. If you don't already know the pytorch API, the code will be pretty obscure. And if you do understand the API, it's still not very clarifying. Instead, it's a lot easier IMO to understand self attention in just plain English. If you understand the words, and also understand the torch API (or tensorflow or numpy, they're all more or less fungible), then it should also be s…

> You have a collection of N things, with some fixed number of features each (doesn't matter how many, but call it d_f). Call that collection {x_i}. You have two learnable matrices Q and K, which learn to project those items into some collection of "questions" and "answers", respectively. Doesn't really matter what those questions or answers are, the NN will figure out what they should be during training. These proje…

Apparently, you're missing the part where they're all defined :)

It does take some getting used to, communicating math in English (and visa-versa).

For example: "two learnable matrices Q and K ... are d_k by d_f". What more do you need to know about them? Also, "a fixed number of features ... call it d_f", I don't know how to more clearly define what d_f is?

I could have called the matrices Joe and Curly, and the feature dimension could have been called Sarah, I suppose. But it's not clear that that's any more helpful... and to anyone with any passing familiarity with math (such as my grad students), giving instances of mathematical objects concise and symbolic names is generally easier to parse, especially when it comes to translating the words into actual math or code.

Re: Revealing example of self-attention, the building block of transformer AI models

#30

Post by cshimmin »

two_in_one wrote: post #25
ntonozzi wrote: post #23

Earlier quoted context omitted.

Thanks for the helpful explanation. What is the context length in this explanation?

my guess it's N here

Correct. In the context of LLM's, the "items" I refer to would be tokens. As a particle physicist, in the transformers I work with the "items" often instead represent either particles, or detector measurements.
Post reply on HN