Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Revealing example of self-attention, the building block of transformer AI models

github.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Revealing example of self-attention, the building block of transformer AI models

#31

Post by fireworkcrayon »

cshimmin wrote: post #29
fireworkcrayon wrote: post #27

Earlier quoted context omitted.

> You have a collection of N things, with some fixed number of features each (doesn't matter how many, but call it d_f). Call that collection {x_i}. You have two learnable matrices Q and K, which learn to project those items into some collection of "questions" and "answers", respectively. Doesn't really matter what those questions or answers are, the NN will figure out what they should be during training. These proje…

Apparently, you're missing the part where they're all defined :) It does take some getting used to, communicating math in English (and visa-versa). For example: "two learnable matrices Q and K ... are d_k by d_f". What more do you need to know about them? Also, "a fixed number of features ... call it d_f", I don't know how to more clearly define what d_f is? I could have called the matrices Joe and Curly, and the fea…

I wasn’t trying to be critical - sorry if it came across that way!

> Also, "a fixed number of features ... call it d_f", I don't know how to more clearly define what d_f is?

Perfect example. I would expect a fixed number to be represented by a single letter, like x. But here, a the fixed number seems to need two letters to represent it: d and f. When I see this, I wonder “what’s d?” “What’s f?” So I scan ahead: I see there is a d_k, so I think there must be some sibling relationship between f and k, but I can’t see what it is. As far as I can tell, Q/q and K/k are the ones that belong in a pair.

It gets very confusing trying to decipher the meaning and relationships.

Re: Revealing example of self-attention, the building block of transformer AI models

#32

Post by GaggiX »

jostmey wrote: post #28
GaggiX wrote: post #22

Earlier quoted context omitted.

If you have a single embedded vector, it should be clear that the self-attention matrix cannot be an arbitrary NxN but only a 1x1, follow the fix I have added to my previous comment. Also if K, Q, V and the self-attention matrix have the same dimensions even when the embed_d and sequence dim have different sizes there is something wrong.

Okay, I think it is fixed. I appreciate your help. If you see this, let me know if the issue still exists!

Now it should be correct.

Re: Revealing example of self-attention, the building block of transformer AI models

#33

Post by cshimmin »

fireworkcrayon wrote: post #31
cshimmin wrote: post #29

Earlier quoted context omitted.

Apparently, you're missing the part where they're all defined :) It does take some getting used to, communicating math in English (and visa-versa). For example: "two learnable matrices Q and K ... are d_k by d_f". What more do you need to know about them? Also, "a fixed number of features ... call it d_f", I don't know how to more clearly define what d_f is? I could have called the matrices Joe and Curly, and the fea…

I wasn’t trying to be critical - sorry if it came across that way! > Also, "a fixed number of features ... call it d_f", I don't know how to more clearly define what d_f is? Perfect example. I would expect a fixed number to be represented by a single letter, like x. But here, a the fixed number seems to need two letters to represent it: d and f. When I see this, I wonder “what’s d?” “What’s f?” So I scan ahead: I see…

No offense taken, I have twice un-deaded your comments because you seem to be posting in good faith :)

Again, you get used to it. Note, the underscore would represent a subscript (in the standard math-writing tool of LaTeX, but also in many variants of Markdown).

Here, d stands for dimension. If you talk about dimensionality a lot, this is one of a few common letters to use to describe it (others popular contenders, especially when speaking about pairs of dimensions would would be n and m, p and q, r and s. But N is already used in a different context [weirdly n/N seems to be an exception to the following paragraph]). And since there are a few different dimensions floating around, we label them all with d, and give them subscripts to indicate which one we're talking about. d_f symbolically means "dimension of the feature space", d_k means "dimension of the key space" (also the same as the query space. in my description I called them question/answer instead of query/key). d_v means "dimension of the value space."

In many contexts, the use of capital letters implicitly denotes matrices. So if you have a matrix that turns x (your input variable), into q (a query vector), naming it Q makes a lot of sense. Especially when we have three matrices floating around, it would have been much more confusing to keep track of if we named the matrices M, N, O. Which one projects to which vector? Much easier to understand that Q makes the q vectors, K makes the k vectors, V makes the v vectors.

Re: Revealing example of self-attention, the building block of transformer AI models

#36

Post by fireworkcrayon »

cshimmin wrote: post #33
fireworkcrayon wrote: post #31

Earlier quoted context omitted.

I wasn’t trying to be critical - sorry if it came across that way! > Also, "a fixed number of features ... call it d_f", I don't know how to more clearly define what d_f is? Perfect example. I would expect a fixed number to be represented by a single letter, like x. But here, a the fixed number seems to need two letters to represent it: d and f. When I see this, I wonder “what’s d?” “What’s f?” So I scan ahead: I see…

No offense taken, I have twice un-deaded your comments because you seem to be posting in good faith :) Again, you get used to it. Note, the underscore would represent a subscript (in the standard math-writing tool of LaTeX, but also in many variants of Markdown). Here, d stands for dimension. If you talk about dimensionality a lot, this is one of a few common letters to use to describe it (others popular contenders,…

Thanks for stepping me through that. I was able to reread and understand your original explanation.

To be honest, though, I was really just trying to understand what my roadblock is in general. As I mentioned, I experience this kind of friction with most math-related writing. And, sadly, I do have at least a "passing familiarity" with math. I don't really struggle with the concepts. For me it's the communication layer: the variety of conventions that exist never seem to jell into a coherent system in the way that natural languages do - even with all of their idiomatic and arbitrary features.

Anyway, thanks again for the discussion.

Re: Revealing example of self-attention, the building block of transformer AI models

#37

Post by 3PS »

cshimmin wrote: post #19

TBH I don't find this exposition enlightening. If you don't already know the pytorch API, the code will be pretty obscure. And if you do understand the API, it's still not very clarifying. Instead, it's a lot easier IMO to understand self attention in just plain English. If you understand the words, and also understand the torch API (or tensorflow or numpy, they're all more or less fungible), then it should also be s…

> Finally, we have a third projection matrix, V, which maps x_j -> v_j. The v_j are d_v-dimensional vectors (doesn't have to equal d_k, and again something you arbitrarily choose). These represent the values that the NN would like to pass on from item j to any other item which decides it "likes" the answer of item j.

> So, to compute the new information to add to item i, we take the weighted sum of the values v_j that item i liked

I don't get this part. If the v_k are d_v dimensional vectors, and if the input items i are all d_f dimensional vectors, then how are you "adding" these values back to the inputs when d_f != d_v?

The linked script doesn't seem to do any adding like this - instead, they take the output values and pass them through a linear layer, presumably throwing away the inputs. But your comment hints at the existence of a more residual sort of approach.

Re: Revealing example of self-attention, the building block of transformer AI models

#38

Post by two_in_one »

cshimmin wrote: post #30
two_in_one wrote: post #25

Earlier quoted context omitted.

my guess it's N here

Correct. In the context of LLM's, the "items" I refer to would be tokens. As a particle physicist, in the transformers I work with the "items" often instead represent either particles, or detector measurements.

Thanks, I'm trying to understand all this mechanics. Matrix multiplication is an equivalent to passing through a single fully connected convolution layer. Which means we can probably beef up and make it a small network. Also it's easy to make it work not in fixed windows, but in scrolling, if the output is processed sequentially. Even if not it may make sense too. It can be implemented efficiently. The idea is that in switching window first and last elements are being influenced only from one side. Which is not a good thing in text processing. Only middle elements are influenced from both sides. With scrolling window all elements currently being processed are in the middle. Except for the ends of the dataset.

Re: Revealing example of self-attention, the building block of transformer AI models

#39

Post by cshimmin »

3PS wrote: post #37
cshimmin wrote: post #19

TBH I don't find this exposition enlightening. If you don't already know the pytorch API, the code will be pretty obscure. And if you do understand the API, it's still not very clarifying. Instead, it's a lot easier IMO to understand self attention in just plain English. If you understand the words, and also understand the torch API (or tensorflow or numpy, they're all more or less fungible), then it should also be s…

> Finally, we have a third projection matrix, V, which maps x_j -> v_j. The v_j are d_v-dimensional vectors (doesn't have to equal d_k, and again something you arbitrarily choose). These represent the values that the NN would like to pass on from item j to any other item which decides it "likes" the answer of item j. > So, to compute the new information to add to item i, we take the weighted sum of the values v_j tha…

The dot product of two d_k-dimensional vectors is a scalar. So the entire NxN matrix qk_ij is just a bunch of scalar numbers. The rows are used to create a weighted sum (i.e. a linear combination) of the d_v-dimensional vectors.
Post reply on HN