The best way to understand transformers is to take Andrej’s Karpathy course on youtube. With a keyboard and a lot of focus time.
Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
201–210 of 240 posts
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#202Suppose someone asked you to complete the sentence: “After I woke up and made breakfast, I drank a glass of …” In America one might say the most likely next words are “orange juice”, or “apple juice” but not “sports car” which has nothing to do with the sentence. Ultimately this is what language models do, given a sequence of data (in this case words) predict the most likely next word(s). For attention, when you read…
When one says "attention is all you need" the implication is that some believe that you need something more than just attention. What is that something which has been demonstrated as unneeded? Is it a theory of how language works?
Recursion. Before transformers attention was used in recurrent neural networks. "attention is all you need" showed that you can just drop the recursion and just use attention, and the outcome is that you get a very nicely parallelizable architechture, allowing more efficient training.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#203I strongly agree with Steve on the potential of Nobel prize for the discovery and I want to add that the team probably going to win it within 10 years. For reference, Jack Kilby won his shared Nobel prize on integrated circuit (IC) that transforms the electronics and computer industry after more than 40 years of the original invention. Here is a very informative video and blog by Google Cloud Tech team explaining the…
There's no nobel prize for Algorthms though. There's no physics involved, unlike the IC. Even if llms turn out to be revolutionary, theres no way a Nobel can be given for Transformers.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#204I'll throw my hat in the ring. A transformer is a type of neural network that, like many networks before, is composed of two parts: the "encoder" that receives a text and builds an internal representation of what the text "means"[1], and the "decoder" that uses the internal representation built by the encoder to generate an output text. Let's say you want to translate the sentence "The train is arriving" to Spanish.…
> The negative side is that it is a computationally inefficient architecture as there are plenty of n^2 operations on the length of the input Is this the reason for the limited token windows?
Yes, kinda. The transformer doesn't have a mechanism for dynamically adjusting its input size, so you need to strike a balance between the window being big enough for practical purposes but also small enough that you can still train the network.
Previous networks with RNNs could in theory receive inputs of arbitrary size, but in practice their performance decreased as the input got longer because they "forgot" the earlier input as they went on. The paper "Neural Machine Translation by Jointly Learning to Align and Translate" solved the forgetting problem by, you guessed it, adding attention to the model.
Eventually people realized that attention was all you needed (ha!), removed the RNN, and here we are.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#205I 'm still confused by the term 'attention', because it implies that something else is actively attending, while it's more about self-similarity. We begin with a sequence of vectors and linearly transform in 3 ways as Q,K and V (these tranformations are learned). The "attention" is (Q.K)*V so "amplify the parts of V according to the similarity between the two other projections Q K". Somehow by doing in parallel a lot of parallel self-similar transformations and stacking them in series we get syntax modeling. It remains a mystery to me what the transformations are supposed to model and why this works so well. This paper might well be one of the most profound discoveries of this century.
Please suggest some paper that delves a bit more into the theory around the architecture.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#206Earlier quoted context omitted.
Compared to rnns... maybe? The big nxn is really a killer. I don't know how to judge parallelizability of different DNN models, you're comparing apples to oranges
When you train a transformer, you're training what the next expected token is. You can train all positions of the sequence each in parallel rather than having to sequentially build up the memory state as you generate the sequence with an LSTM. Mind you the inference portion of a transformer is still sequentially bottlenecked since you don't know what the output sequence is supposed to be.
If you train in parallel, how do you combine the weights generated from different trainings in parallel?
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#207Not ELI5 obviously but might help some.
Transformer is a patterning probabilistic machine for a sequence of identities[1]. These identities are fed to the transformer in lanes. The transformer is conditioned to shift lanes one position to the left until they make it to the output, and make a prediction in the right-most lane that got freed up. Attention adds an exponential amount of layer interconnectivity, when we compare it with a simple densely connected layers. The attention mask serves as a high-dimensional dropout, without which it would be extremely easy for the Transformer to simply repeat the inputs (and then fail to generalize when making the prediction). Each layer up until the vertical middle of the Transformer works with a higher contextual representation than the previous one, and this is again being unwound back to lower contexts from the middle layer back to the original identities (integers) on the outputs. This means that you have raw identities on the input and output which span a certain width/window of the input sequence, but in comparison the middle-most layer has a sequence of high level contexts spanning extreme lengths of the original input sequence, knowledge-wise. [1]It's important to know that modification (learning) by the Transformer, of the vector embeddings which represent the input/output identities/integers that the Transformer works with, constitute big portion of the Transformer's power, and the practical implication of that is that it's impractical to try to tell the Transformer that e.g. some of our identities are similar or there's some logical system in their similarity, because all the Transformer really cares about is the occurrence of these identities in the sequence we train the Transformer on, and the Transformer will figure out the similarities or any kind of logic in the sequence by itself.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#208i dont get it. they tried an architecture, then couldn't be bothered to describe it well...
i remain disappointed at the staggering low quality of academic work. the writing here is appalling. no worked example provided, despite of evidence of one... typical academic crap.
wouldn't surprise me if you go to try it and its wrong, and none of the real problems have been solved.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#209i dont get it. they tried an architecture, then couldn't be bothered to describe it well... i remain disappointed at the staggering low quality of academic work. the writing here is appalling. no worked example provided, despite of evidence of one... typical academic crap. wouldn't surprise me if you go to try it and its wrong, and none of the real problems have been solved.
I do not understand what exactly you are complaining about.
It is true that the "Attention Is All You Need"[0] paper requires some level of basic understanding/experience of ML architectures and engineering, but the target audience for the paper certainly understands it. It would be unnecessary if every academic paper startet with a first principles explanation of what is considered common knowledge in the field.
> wouldn't surprise me if you go to try it and its wrong, and none of the real problems have been solved
But obviously attention based transformer architectures are solving real world problems (by being better than previous architectures in real world applications).
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#210Sure! Transformers are a type of deep learning model that have revolutionized the field of natural language processing (NLP) and achieved state-of-the-art performance on various tasks like machine translation, text summarization, and question answering.
The "Attention is All You Need" paper, published in 2017 by Vaswani et al., introduced the Transformer model architecture. The paper proposed a new way to process sequences of data, such as words in a sentence or time steps in a time series, without using recurrent neural networks (RNNs) or convolutional neural networks (CNNs). Instead, it introduced a mechanism called "self-attention."
Self-attention allows the model to weigh the importance of different words in a sentence when processing each word. This attention mechanism helps the model to focus on the relevant parts of the input sequence. In other words, it pays attention to different words based on their contextual significance for a given task.
To understand self-attention, let's consider an example. Suppose we have a sentence: "The cat sat on the mat." When processing the word "sat," self-attention enables the model to assign higher weights to words like "cat" and "the" and lower weights to words like "on" and "the mat." This way, the model can learn which words are more relevant to understanding the context of "sat."
The Transformer model consists of an encoder and a decoder. The encoder processes the input sequence, such as a sentence, while the decoder generates the output sequence, such as a translated sentence. Both the encoder and decoder are composed of multiple layers of self-attention and feed-forward neural networks. The self-attention layers allow the model to capture dependencies between different words in the sequence, while the feed-forward networks help in learning more complex patterns.
The "Attention is All You Need" paper demonstrated that Transformers achieved state-of-the-art performance on machine translation tasks while being more parallelizable and requiring less training time compared to traditional RNN-based models. Since then, Transformers have become the go-to architecture for many NLP tasks and have been further improved with variations like BERT, GPT, and T5.
In summary, the Transformer model introduced in the "Attention is All You Need" paper replaced traditional recurrent or convolutional neural networks with self-attention, allowing the model to capture contextual relationships between words more effectively. This innovation has had a significant impact on the field of NLP and has become the foundation for many subsequent advances in the field.