Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

Yeah sorry, it still requires math and probably some exposure to ML basics.

It was never meant to be taken literally, it's just an idiomatic way of asking that something be explained as simply as possible.

An answer to OP's question targeted at a five year old level would likely be impossible given the subject matter, or else need to be so general and simplistic as to be useless on this forum of not actual five year olds.

I think one hole in the description for simplicity is that "differentiable" it's not an adjective that applies to hash tables.

Differentiable relative to what? What is (x) in the d(hashtable)/d(x) equation?

Explaining it for a slightly older audience, a transformer is a type of artificial neural network designed for processing sequences, like sentences in a text. It's especially known for its use in natural language processing (NLP), which is the field of AI that deals with understanding and generating human language.

The Transformer is unique because it uses a mechanism called "attention" to understand the relationships between words in a sentence, which works like this:

(1) Encoding: First, the Transformer turns each word in a sentence into a list of numbers, called a vector. These vectors capture information about the word's meaning.

(2) Self-Attention: Next, for each word, the Transformer calculates a score for every other word in the sentence. These scores determine how much each word should contribute to the understanding of the current word. This is the "attention" part. For example, in the sentence "The cat, which is black, sat on the mat," the words "cat" and "black" would get high scores when trying to understand the word "black" because they are closely related.

(3) Aggregation: The Transformer then combines the vectors of all the words, weighted by their attention scores, to create a new vector for each word. This new vector captures both the meaning of the word itself and the context provided by the other words in the sentence.

(4) Decoding: Finally, in a task like translation, the Transformer uses the vectors from the encoding phase to generate a sentence in the target language. It again uses attention to decide which words in the original sentence are most relevant for each word it's trying to generate in the new sentence.

One key advantage of the Transformer is that it can calculate the attention scores for all pairs of words at the same time, rather than one at a time like previous models. This allows it to process sentences more quickly, which is important for large tasks like translating a whole book.

Thanks a lot for sharing. I liked the explanation, although it seems to be not a perfect matching between attention and kernel (as far as I understand). Since as I undestand x_o = Qx and x_i = Qx and y_i = Vx but it doesn't map to Wu dot Wv.

Anyway just curious do you or someone else have more sources in this format?

At least this way it is very helpfull to think about the methods. Kind of agree that the formular does look very very similar to well known methods. But on the other hand, the author doesn't explain the transformers in a similar simple way since it is not obvious why you could stack kernels and get better results.