Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 240 posts

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#161

Post by vitno »

rshm wrote: post #154
UncleOxidant wrote: post #150

Earlier quoted context omitted.

Someone who read the paper pointed out to me recently that there's an aspect to transformers/attention that uses the sin or cos function to determine which words to pay attention to or the spacing between them (I'm probably not expressing this correctly, so please correct me if I'm wrong). It seems really unintuitive that sin and/or cos would be a factor in human language - can you explain this?

Someone else can better explain this. Based on one of the video suggested in one of the replies here. sin and cos doesn’t have any inherent properties specific to language, they were chosen because a simple linear function was needed in that step of optimization. Any other function could fit the bill as well.

Sin or cosine was chosen explicitly because they are non-linear functions

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#162

Post by moultano »

shadowgovt wrote: post #106
throwawaymaths wrote: post #102

Earlier quoted context omitted.

Yeah sorry, it still requires math and probably some exposure to ML basics.

I think one hole in the description for simplicity is that "differentiable" it's not an adjective that applies to hash tables. Differentiable relative to what? What is (x) in the d(hashtable)/d(x) equation?

differentiable relative to the model parameters. The attention mechanism does store(key(x), value(x)), followed by lookup(query(y)), where key(), value(), lookup(), query() are all designed to be differentiable.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#163

Post by armchairhacker »

mahathu wrote: post #138
rosebay wrote: post #113

Earlier quoted context omitted.

Great explanation

It's chat-gpt generated. It's even leaked part of the prompt in the intro. I especially disagree with: > natural language processing (NLP), which is the field of AI that deals with understanding and generating human language.

If GPT4 wrote this it did a great job, and highlights how incredible and useful it can be.

Although I'm sure "ELI5 what is a transformer" was one of the RHLF prompts which got handcrafted responses from OpenAI engineers whose bread and butter is transformers, so...still a great response.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#164

Post by mensetmanusman »

data_maan wrote: post #47

What is ELI5? The idea behind the Transformer is nice - but by far not Nobel prize deserving. Don't believe the hype or people like Yegge, whoever that is - in a few years a new architecture will be the "Nobel candidate". Also, the original Transformer paper, if you read is, is horribly written.

Something seems Nobel related for the tech to be so useful.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#165

Post by LudwigNagasena »

UncleOxidant wrote: post #150
QuantumGood wrote: post #108

Earlier quoted context omitted.

Explaining it for a slightly older audience, a transformer is a type of artificial neural network designed for processing sequences, like sentences in a text. It's especially known for its use in natural language processing (NLP), which is the field of AI that deals with understanding and generating human language. The Transformer is unique because it uses a mechanism called "attention" to understand the relationship…

Someone who read the paper pointed out to me recently that there's an aspect to transformers/attention that uses the sin or cos function to determine which words to pay attention to or the spacing between them (I'm probably not expressing this correctly, so please correct me if I'm wrong). It seems really unintuitive that sin and/or cos would be a factor in human language - can you explain this?

It is explained in the paper.

> In this work, we use sine and cosine functions of different frequencies:

> PE(pos,2i) = sin(pos/10000^{2i/d_model})

> PE(pos,2i+1) = cos(pos/10000^{2i/d_model})

> where pos is the position and i is the dimension. That is, each dimension of the positional encoding corresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000 · 2π. We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PE_{pos+k} can be represented as a linear function of PE_{pos}.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#166

Post by fwlr »

UncleOxidant wrote: post #150
QuantumGood wrote: post #108

Earlier quoted context omitted.

Explaining it for a slightly older audience, a transformer is a type of artificial neural network designed for processing sequences, like sentences in a text. It's especially known for its use in natural language processing (NLP), which is the field of AI that deals with understanding and generating human language. The Transformer is unique because it uses a mechanism called "attention" to understand the relationship…

Someone who read the paper pointed out to me recently that there's an aspect to transformers/attention that uses the sin or cos function to determine which words to pay attention to or the spacing between them (I'm probably not expressing this correctly, so please correct me if I'm wrong). It seems really unintuitive that sin and/or cos would be a factor in human language - can you explain this?

Not as much of an expert as others commenting here, but I believe the sine/cosine stuff comes in just because it’s a standard and very efficient way of comparing vectors.

(“Vector” is just an alternate way of talking about a coordinate point - you can say “a point is at (x,y)”, or equivalently you can also say “turn x degrees and then travel y units forward”, either method gives enough information to find the point exactly.)

I don’t think sine and cosine are actually factors in human language - rather, the process of turning words into vectors captures whatever are the factors in human language, translates them into vectors, and in that translation process the factors get turned into something that sine/cosine measurements of vectors is good at picking up.

A toy example would be that arithmetic doesn’t seem to be a factor in political orientation, but if you assess everyone’s political orientation with some survey questions and then put them on a line from 0 to 10, then you could do some subtraction and multiplication operations to find numbers that are close together - ie doing arithmetic to find similar politics. The reason that works is not because arithmetic has anything to do with political orientation, it’s because your survey questions captured information about political orientation and transformed it into something that arithmetic works on.

I guess this explanation doesn’t do much except push the unintuitiveness into the embedding process (that’s the process of turning words into vectors in a way that captures their relationship to other words).

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#168

Post by nborwankar »

sn41 wrote: post #156
QuantumGood wrote: post #108

Earlier quoted context omitted.

Explaining it for a slightly older audience, a transformer is a type of artificial neural network designed for processing sequences, like sentences in a text. It's especially known for its use in natural language processing (NLP), which is the field of AI that deals with understanding and generating human language. The Transformer is unique because it uses a mechanism called "attention" to understand the relationship…

complete newbie here: what is the intuition behind the conclusion that "cat" is highly related to "black" as opposed to, say, "mat"?

Attention and the Transformer make it possible to recognize that the probability of “black” applying to the cat is much much higher than to the mat due to the phrasing “which is” in between the cat and black.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#169

Post by djbusby »

aGHz wrote: post #160
sn41 wrote: post #156

Earlier quoted context omitted.

complete newbie here: what is the intuition behind the conclusion that "cat" is highly related to "black" as opposed to, say, "mat"?

It is a lot harder to take the black out of the cat than it is to take the mat out from under it.

Humans know that, how does transform know that? Based on training data?

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#170

Post by hackernewds »

benjismith wrote: post #90

Okay, here's my attempt! First, we take a sequence of words and represent it as a grid of numbers: each column of the grid is a separate word, and each row of the grid is a measurement of some property of that word. Words with similar meanings are likely to have similar numerical values on a row-by-row basis. (During the training process, we create a dictionary of all possible words, with a column of numbers for each…

why are the words cols and properties are rows. seems counter intuitive
Post reply on HN