Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 240 posts

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#171

Post by cwillu »

soVeryTired wrote: post #148
QuantumGood wrote: post #147

Earlier quoted context omitted.

In the context of natural language processing, the attention mechanism used in Transformer models and the process of converting tokens to vectors and calculating cosine similarity have similarities but serve different purposes. When you convert words (tokens) into vectors and calculate cosine similarity, you're typically doing what's called "word embedding". This process captures the semantic meaning of words in a hi…

Just a guess: is this answer GPT output?

From another one of his responses that he has since deleted from this thread: “Despite these challenges, researchers have found that hierarchical attention can improve performance on tasks like document classification, summarization, and question answering, especially for longer texts. As of my last training cut-off in September 2021, this is an area of ongoing research and development.”

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#172

Post by dominickramer »

Suppose someone asked you to complete the sentence:

“After I woke up and made breakfast, I drank a glass of …”

In America one might say the most likely next words are “orange juice”, or “apple juice” but not “sports car” which has nothing to do with the sentence.

Ultimately this is what language models do, given a sequence of data (in this case words) predict the most likely next word(s).

For attention, when you read the sentence, which words stood out as more important? Probably woke up, breakfast, and glass while the words after, I, and made were less important to completing the sentence.

That is, you paid more attention to the important words to understand how to complete the sentence.

The “attention mechanism” in language models is a way to let the models learn which words are important in sentences and pay more attention to them too when completing sentences, just like a person would do as in the example above.

Further, it turns out this attention mechanism lets the models do lots of interesting things even without other fancy model techniques. That is “attention is all you need”.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#173

Post by sn41 »

nborwankar wrote: post #168
sn41 wrote: post #156

Earlier quoted context omitted.

complete newbie here: what is the intuition behind the conclusion that "cat" is highly related to "black" as opposed to, say, "mat"?

Attention and the Transformer make it possible to recognize that the probability of “black” applying to the cat is much much higher than to the mat due to the phrasing “which is” in between the cat and black.

Thank you. So this is based on the training data, I assume.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#175

Post by Accujack »

djbusby wrote: post #169
aGHz wrote: post #160

Earlier quoted context omitted.

It is a lot harder to take the black out of the cat than it is to take the mat out from under it.

Humans know that, how does transform know that? Based on training data?

Sort of. Part of the training for a model includes telling it which parts of a sentence are important... a human points and clicks.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#176

Post by panarky »

QuantumGood wrote: post #108
QuantumGood wrote: post #99

The "Attention is All You Need" paper introduced a new way for AI to read and understand language, much like how we might read a comic book. As you read each panel of a comic book, you don't just look at the words in the speech bubbles, but you also pay attention to who's talking, what they're doing, and what happened in the previous panels. You might pay more attention to some parts than others. This is sort of like…

Explaining it for a slightly older audience, a transformer is a type of artificial neural network designed for processing sequences, like sentences in a text. It's especially known for its use in natural language processing (NLP), which is the field of AI that deals with understanding and generating human language. The Transformer is unique because it uses a mechanism called "attention" to understand the relationship…

> each word in a sentence ...

Does each sentence stand alone, or is the meaning of the sentence, and the words in the sentence, influenced by the sentences that come before and after it?

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#177

Post by andreyk »

QuantumGood wrote: post #135
QuantumGood wrote: post #108

Earlier quoted context omitted.

Explaining it for a slightly older audience, a transformer is a type of artificial neural network designed for processing sequences, like sentences in a text. It's especially known for its use in natural language processing (NLP), which is the field of AI that deals with understanding and generating human language. The Transformer is unique because it uses a mechanism called "attention" to understand the relationship…

The importance of the "Attention is All You Need" paper by Vaswani et al., in 2017 is that it introduced the Transformer type of model architecture. The model is so named because it "transforms" one sequence into another. For example, in a machine translation task, it can transform a sentence in one language into a sentence in another language. The key innovation of the Transformer model is the use of self-attention…

Just a quick clarification, attention of the same sort transformers used was already being employed in RNNs for a while. Thus the name "attention is all you need", it turned out you can just remove the recurrent part which makes it hard to train the NN.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#178

Post by elankart »

This might be dumbing this down very very much but it all boils down to having some sort of a special lookup table and you give it a "query" and a bunch of "keys" and the "value" is the most likely next word. As you input something into the network, a network of these tables are consulted and you are given most likely next word.

The novelty in this paper is this "query-key-value" relation that gets learned. A lot of previous work in this area was focused on learning a rough state machine to which you input a set of state transitions and it will give you the most likely next state. This will also work but training such networks is very slow and you also don't have the capability to train the network to "attend" to certain part of the inputs. This lookup based technique lets you do that plus this is also very compute efficient (compared to previous techniques).

I'm missing a lot of details but that's basically the intuition behind this.

These are very excellent resources: - https://www.youtube.com/watch?v=ptuGllU5SQQ&list=PLoROMvodv4... - https://www.youtube.com/watch?v=OyFJWRnt_AY&pp=ygUfYXR0ZW50a...

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#179

Post by TaurenHunter »

UncleOxidant wrote: post #150
QuantumGood wrote: post #108

Earlier quoted context omitted.

Explaining it for a slightly older audience, a transformer is a type of artificial neural network designed for processing sequences, like sentences in a text. It's especially known for its use in natural language processing (NLP), which is the field of AI that deals with understanding and generating human language. The Transformer is unique because it uses a mechanism called "attention" to understand the relationship…

Someone who read the paper pointed out to me recently that there's an aspect to transformers/attention that uses the sin or cos function to determine which words to pay attention to or the spacing between them (I'm probably not expressing this correctly, so please correct me if I'm wrong). It seems really unintuitive that sin and/or cos would be a factor in human language - can you explain this?

That sounds like a reference to the concept of cosine similarity.

Imagine that words are spread out in the space. Cosine similarity is a measure of similarity between two vectors (each word is encoded as a vector).

By measuring the cosine of the angle between the two vectors we can get:

1) whether 2 vectors have the same angle (2 words have the same meaning or close enough) when the cosine is close to 1

2) whether 2 vectors are perpendicular (2 words don't have anything to do with each other) when the cosine is close to zero

3) whether 2 vectors are opposite in direction (2 words have opposite meanings in some aspect) when the cosine is close to -1

Cosine similarity is like comparing two people's interests. If two people have similar interests, the angle between them is small, and the cosine similarity value will be high. If two people have completely different interests, the angle between them is large, and the cosine similarity value will be low. So, cosine similarity is a way to measure how similar two things are by looking at the angle between them.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#180

Post by testrun »

Accujack wrote: post #175
djbusby wrote: post #169

Earlier quoted context omitted.

Humans know that, how does transform know that? Based on training data?

Sort of. Part of the training for a model includes telling it which parts of a sentence are important... a human points and clicks.

This is extremely important to know. That the relationships between words in the sentence are actually trained by human evaluation.
Post reply on HN