Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

"Transformers" and "Attention is All You Need" refer to an important development in machine learning and artificial intelligence, particularly in the field of natural language processing (NLP). I'll try to explain them in a simple way.

Think of a conversation you had with a friend. While they were talking, you were probably not just listening to the words they were saying right now, but also remembering what they said a few minutes ago. Your brain was connecting the dots between different parts of the conversation to understand the full meaning. Now, imagine if you could only understand each word in isolation and couldn't remember anything from a few seconds ago. Conversations would be pretty hard to understand, right?

In early NLP models, this was a big problem. They couldn't easily look at the "context" of a conversation or a sentence. They could only look at a few words at a time, so they were a bit like our forgetful person. They were good at understanding the meaning of individual words, but not so good at understanding how those words fit together to create meaning.

I came here to post this video. It’s a great primer on the topic and it gives you ideas to prompt gpt and have it output more.

It’s how I got an understanding of beam search, a technique employed in some of the response building.

Asked to provide some code samples:

""" Imagine you have a sentence: "The cat sat on the mat." In traditional models, each word is processed one by one, and the model might miss important relationships between words that are far apart. Transformers solve this problem using self-attention.

In self-attention, the model pays attention to all the words in the sentence simultaneously, giving more importance to different words depending on their relevance to each other. Here's a code snippet to illustrate self-attention:

    import torch
    import torch.nn as nn

    # Assume we have an input sentence with 5 words
    input_sentence = torch.tensor([[0.1, 0.2, 0.3, 0.4, 0.5]])

    # Define the self-attention layer
    self_attention = nn.MultiheadAttention(embed_dim=5, num_heads=1)

    # Apply self-attention to the input sentence
    attention_output, _ = self_attention(input_sentence, input_sentence, input_sentence)

    # The attention_output represents the transformed sentence with attention
    print(attention_output)

In this code, we use a simple example with one self-attention head. The input sentence is represented as a tensor with shape (1, 5), where each number represents the word's features. We pass the input sentence to the self-attention layer twice, using it as the query, key, and value inputs.

The self-attention layer computes the attention scores between each word in the sentence, determining how much attention each word should receive from the other words. It then combines the input sentence with these attention weights to produce the attention_output.

The attention_output contains the transformed sentence, where each word now carries information from other words based on their importance. This allows the model to capture relationships and dependencies between words effectively.

The "Attention is All You Need" paper introduced this self-attention mechanism as a fundamental building block in the Transformer model. The Transformer architecture then incorporates multiple layers of self-attention, along with other components like positional encoding, to process sequences and achieve impressive results in various NLP tasks.

Remember, this code snippet is just a simplified illustration to help you understand the concept of self-attention. The actual implementation of Transformers is more complex, involving multiple layers, feed-forward networks, and additional optimizations. """

The nn.MultiheadAttention documentation [1], the output of print(attention_output) is:

    tensor([[ 0.1756, -0.2273, -0.0787,  0.0383, -0.0779]], grad_fn=)

If you badger ChatGPT it will give you an example with different query, key, and value inputs

    # Define distinct query, key, and value inputs
    query = torch.tensor([[0.2, 0.3, 0.4, 0.5, 0.6]])
    key = torch.tensor([[0.3, 0.4, 0.5, 0.6, 0.7]])
    value = torch.tensor([[0.4, 0.5, 0.6, 0.7, 0.8]])

    # Apply self-attention to the input sentence using distinct inputs
    attention_output, _ = self_attention(query, key, value)

outputting:

    tensor([[-0.2294,  0.1336, -0.1445, -0.7275, -0.1232]], grad_fn=)

Pushed it then further to explain how does nn.MultiheadAttention works:

""" In PyTorch, nn.MultiheadAttention is a module that implements the multi-head attention mechanism. Multi-head attention is a key component of the Transformer architecture, which is widely used in various natural language processing (NLP) tasks.

The nn.MultiheadAttention module allows you to apply multi-head attention to a set of queries, keys, and values. Here's a simplified explanation of what it does:

Inputs: The module takes three inputs: queries, keys, and values. These inputs are typically tensors that represent the encoded representations of the input sequence.

Linear Projections: Before applying attention, the module performs linear projections on the queries, keys, and values. This step projects the input tensors into different subspaces to capture different aspects of the data. These projections are learnable parameters.

Attention Scores: Next, the module computes attention scores between the queries and the keys. The attention scores measure the relevance or importance of each key with respect to each query. Higher attention scores indicate a stronger relationship between the query and the corresponding key.

Attention Weights: The attention scores are transformed into attention weights by applying a softmax function. The softmax operation normalizes the attention scores, ensuring that the weights sum up to 1 and represent a valid distribution.

Weighted Sum: The attention weights are used to compute a weighted sum of the values. This step aggregates the values based on their importance, as determined by the attention weights. The weighted sum produces the final output of the attention mechanism.

Multi-Head Attention: The module performs the above steps multiple times in parallel, each time using different sets of learned linear projections. These parallel attention operations are called "attention heads." The outputs of the attention heads are concatenated and linearly transformed to obtain the final output of the nn.MultiheadAttention module.

The multi-head attention mechanism allows the model to capture different types of relationships and dependencies in the input sequence. By using multiple attention heads with different linear projections, the model can attend to different parts of the input space and learn diverse representations.

Overall, nn.MultiheadAttention in PyTorch provides a convenient way to incorporate the multi-head attention mechanism into your neural network architecture, facilitating effective modeling of complex dependencies in sequence data. """

[1] https://pytorch.org/docs/stable/generated/torch.nn.Multihead...

It’s not really something you need to understand unless you’re an ML researcher.

I guess the ELI5 (with a BUNCH of details left out) is “Transformers: what if you didn’t have to process sentences as a sequence of words, but rather as a picture of words.”

From the Yegge post:

> LoRA makes LLMs composable, piecewise, mathematically, so that if there are 10,000 LLMs in the wild, they will all eventually converge on having the same knowledge. This is what Geoffrey Hinton was referring to on his SkyNet tour.

I don't think that's right at all, LoRA freeze lots of the large model part and wouldn't let you just simply combine large models. Instead. I'm pretty sure Hinton is referring to data parallel training with batching:

> DataParallel (DP) - the same setup is replicated multiple times, and each being fed a slice of the data. The processing is done in parallel and all setups are synchronized at the end of each training step.

https://huggingface.co/docs/transformers/v4.15.0/parallelism

You can have many instances of the model training on different bits of data, and then just average the modified weights back together at the end. This combining of weights is what Hinton means when he says parallel copies of brains can learn things independently and then recombine them later a huge bandwidth speeds, whereas humans are far more limited to sharing separate experiences verbally or with like a multimedia presentation or something.