231–240 of 240 posts
Okay, here's my attempt! First, we take a sequence of words and represent it as a grid of numbers: each column of the grid is a separate word, and each row of the grid is a measurement of some property of that word. Words with similar meanings are likely to have similar numerical values on a row-by-row basis. (During the training process, we create a dictionary of all possible words, with a column of numbers for each…
Okay, here's my attempt! First, we take a sequence of words and represent it as a grid of numbers: each column of the grid is a separate word, and each row of the grid is a measurement of some property of that word. Words with similar meanings are likely to have similar numerical values on a row-by-row basis. (During the training process, we create a dictionary of all possible words, with a column of numbers for each…
I have a very dumb question, I'll just throw it here: I understand word embeddings and tokenisation- and the value of each; but how can the two work together? Are embeddings calculated for tokens, and in that case, how useful are they, given that each token is just a fragment of a word, often with little or no semantic meaning?
Decoder-only architectures (such as GPT) mask the token embedding interaction matrix (attention) such each token embedding and all subsequent transformations only have access to preceeding token embeddings (and transforms). This means that on output, only the last transformed token embedding has the full information of the entire context - and only it is capable of making predictions for the next token.
This is done so that during training, you can simultaneously make 1000s (context size) of predictions - every final token embedding transform is predicting the next token. The alternative (Encoder architecture, where there is no masking and the first token can interact with the final token) would result in massively inefficient training for predicting the next token as each full context can only make a single prediction.
The Yannic kilcher review is quite good. https://youtu.be/iDulhoQ2pro I can't ELI5 but I can ELI-junior-dev. Tl;dw: Transformers work by basically being a differentiable lookup/hash table. First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame. Then there is an NxN matrix that is applied to your attention frame "…
What does it mean for a lookup/hash table to be differentiable?
It's differentiable because how much of a fraction to grab is the result of a simple dot product followed by a softmax.
Sure! Transformers are a type of deep learning model that have revolutionized the field of natural language processing (NLP) and achieved state-of-the-art performance on various tasks like machine translation, text summarization, and question answering. The "Attention is All You Need" paper, published in 2017 by Vaswani et al., introduced the Transformer model architecture. The paper proposed a new way to process seq…
You know what would be great to add to HN? A chatGPT response below each comment. So nobody else finds the need to do this.
Earlier quoted context omitted.
I have a very dumb question, I'll just throw it here: I understand word embeddings and tokenisation- and the value of each; but how can the two work together? Are embeddings calculated for tokens, and in that case, how useful are they, given that each token is just a fragment of a word, often with little or no semantic meaning?
The way transformers operate is by transforming the embedding space through each layer. You could say that all the "understanding" is happening in that high dimensional space - that of a single token, but multiplied by the number of tokens. Seeding the embedding space with some learned value for each token is helpful. Think of it as just a vector database: token -> vector. Decoder-only architectures (such as GPT) mas…
Marqo supports E5 models: https://github.com/marqo-ai/marqo
Here is an attempt that sacrifices technical precision for intuition. We're going to represent words as vectors (a sequence of numbers). We would like it to be the case that the value of the numbers reflects the meaning of the words. Words that mean similar things should be near each other. We also want to represent higher level ideas, ideas that take multiple words to express, in the same way. You can think of all t…
Thank you for this explanation. I've found that the QKV concepts are some of the most glossed over parts of attention and I'll be honest, some of the most confusing. Would you mind actually going into more detail on the questions you asked towards the end of your post? I vaguely understand how an embedding might get learned over time, but I don't understand how queries are "constructed" or how these three separate ma…
Earlier quoted context omitted.
When one says "attention is all you need" the implication is that some believe that you need something more than just attention. What is that something which has been demonstrated as unneeded? Is it a theory of how language works?
Recursion. Before transformers attention was used in recurrent neural networks. "attention is all you need" showed that you can just drop the recursion and just use attention, and the outcome is that you get a very nicely parallelizable architechture, allowing more efficient training.
The "Attention is All You Need" paper introduced a new way for AI to read and understand language, much like how we might read a comic book. As you read each panel of a comic book, you don't just look at the words in the speech bubbles, but you also pay attention to who's talking, what they're doing, and what happened in the previous panels. You might pay more attention to some parts than others. This is sort of like…
Please don't post generated comments into HN threads. HN threads are for human discussion and we ban accounts that violate this.