1–10 of 240 posts
Makes me want to better understand this tech.
Edit: thank you for some amazing top level responses and links to valuable content on this subject.
Transformers, Explained: Understand the Model Behind GPT-3, BERT, and T5: https://daleonai.com/transformers-explained
Transformers from Scratch: https://e2eml.school/transformers.html
The first link covers Attention well.
I worked on a few projects that were trying to develop foundation models for health care, aviation, and other domains. In particular I trained an LSTM model to write fake abstracts for clinical case reports.
We ran into many problems, but maybe the worst one with the LSTM is that a real document repeats itself a lot. For instance, somebody's name might turn up multiple times and the LSTM was very bad at that kind of copying. The LSTM community was arguing about solutions to this problem, but the attention mechanism in transformers makes it easy.
[flagged]
Sure! Imagine you're in school, and your teacher asks you and your friends to work on a group project. Each of you has a different set of skills, and you need to work together to complete the project successfully. In this analogy, the group project is like a sentence or a piece of text that a language model is trying to understand.
Transformers are a type of model that helps computers understand and generate text, like in our group project. The key idea behind transformers is something called "attention." Attention helps the model figure out which words in a sentence are the most important to focus on, just like you and your friends pay attention to each other's skills to complete your project.
The "Attention is all we need" paper introduced transformers and showed that by using attention, the model can learn how to understand and generate text more effectively. Instead of focusing on each word one at a time, transformers can look at all the words together, decide which ones are the most important, and use that information to understand the text better.
So, in our group project example, transformers help the model work like a team, where everyone pays attention to each other's strengths and helps each other out. This makes the model more powerful and able to do a better job at understanding and generating text.
[flagged]
"Imagine you have a really smart computer friend that can understand and talk to you just like a person. This computer friend is called a language transformer.
The language transformer is like a super good reader. It reads lots and lots of books, stories, and articles to learn how people talk and write. It pays attention to how words go together and what they mean in different situations. It becomes really good at understanding what people are saying or asking.
Once the language transformer has learned so much, it can help with many things. For example, if you want to write a story, it can help you come up with ideas and write sentences that make sense. If you have a question, it can try to find the answer for you. It can also help translate words from one language to another, like when you want to understand what someone is saying in a different language.
The language transformer is like a big brain that knows a lot about words and how they fit together. It uses all that knowledge to talk and help you with different things."
I can't ELI5 but I can ELI-junior-dev. Tl;dw:
Transformers work by basically being a differentiable lookup/hash table. First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame.
Then there is an NxN matrix that is applied to your attention frame "performing the lookup query" over all other tokens in the attention frame, so every token gets a "contextual semantic understanding" that takes in both all the other stuff in the attention frame and it's relative position.
Gpt is impressive because the N is really huge and it has many layers. A big N means you can potentially access information farther away. Each layer gives more opportunities to summarize and integrate long range information in a fractal process.
Two key takeaways:
- differentiable hash tables
- encoding relative position using periodic functions
NB: the attention frame tokens are actually K-vectors (so the frame is a KxN matrix) and the query matrix is an NxNxK tensor IIRC but it's easier to describe it this way
Basically tokens “talk” to each other and say this is what i have and this is what i look for.
An attention mechanism is when you want a neural network to learn the function of how much attention to allocate to each item in a sequence, to learn which items should be looked at.
Transformers is a self-attention mechanism, where you ask the neural network to 'transform' each element by looking at its potential combination with every other element and using this (learnable, trainable) attention function to decide which combination(s) to apply.
And it turns out that this very general mechanism, although compute-intensive (it considers everything linking with everything, so complexity quadratic to sequence length) and data-intensive (it has lots and lots of parameters, so needs huge amounts of data to be useful) can actually represent many of things we care about in a manner which can be trained with the deep learning algorithms we already had.
And, really, that's the two big things ML needs, a model structure where there exists some configuration of parameters which can actually represent the thing you want to calculate, and that this configuration can actually be determined from training data reasonably.