31–40 of 181 posts
Transformer is hard to describe with analogies, and TBF there is no good explanation why it works, so it may be better to just present the mechanism, "leaving the interpretation to the viewer". Also, it's simpler to describe dot products as vectors projecting on one another
Awesome video. This helps to show how the Q*K matrix multiplication is a bottleneck, because if you have sequence (context window) length S, then you need to store an SxS size matrix (the result of all queries times all keys) in memory. One great way to improve on this bottleneck is a new-ish idea called Ring Attention. This is a good article explaining it: https://learnandburn.ai/p/how-to-build-a-10m-token-context (…
I think what made this so digestible for me were the animations. The timing, how they expand/contract and unfold while he’s speaking.. is all very well done.
Awesome video. This helps to show how the Q*K matrix multiplication is a bottleneck, because if you have sequence (context window) length S, then you need to store an SxS size matrix (the result of all queries times all keys) in memory. One great way to improve on this bottleneck is a new-ish idea called Ring Attention. This is a good article explaining it: https://learnandburn.ai/p/how-to-build-a-10m-token-context (…
In Unsloth, memory usage scales linearly (not quadratically) due to Flash Attention (+ you get 2x faster finetuning, 80% less VRAM use + 2x faster inference). Still O(N^2) FLOPs though.
On that note, on long contexts, Unsloth's latest release fits 4x longer contexts than HF+FA2 with +1.9% overhead. So 228K context on H100.
Hold on, every predicted token is only a function of the previous token? I must have something wrong. This would mean that within the embedding of "was", which is of length 12,228 in this example. Is it really possible that this space is so rich as to have a single point in it encapsulate a whole novel?
it's not about a single point encapsulating a novel, but how sequences of such embeddings can represent complex ideas when processed by the model's layers. each prediction is based on a weighted context of all previous tokens, not just the immediately preceding one.
I suppose that when you each element in the vector weighs 16 bits then the space is immense and capable to have a novel in a point.
I finally understand this! Why did every other video make it so confusing!
It is confusing, 3b1b is just that good.
attention(Q,K,V) = softmax (Q K^T √ dK ) @ V
is just half a row; the multi-head, masking and positional stuff just toppings
we have many basic algorithms in CS that are more involved, it's amazing we get language understanding from such simple math
Hold on, every predicted token is only a function of the previous token? I must have something wrong. This would mean that within the embedding of "was", which is of length 12,228 in this example. Is it really possible that this space is so rich as to have a single point in it encapsulate a whole novel?
at that point what it has is not a representation of the input, its a representation of what the next output could be. ie. its a lossy process and you can't extract what came in the past, only the details relevant to next word prediction (is my understanding)
I finally understand this! Why did every other video make it so confusing!
Earlier quoted context omitted.
Because: 1. good communication requires an intelligence that most people sadly lack 2. because the type of people who are smart enough to invent transformers have zero incentive to make them easily understandable. most documents are written by authors subconsciously desperate to mentally flex on their peers.
Pedagogy requires empathy, to know what it's like to not know something. They'll often draw on experiences the listener is already familiar with, and then bridge the gap. This skill is orthogonal to the mastery of the subject itself, which I think is the reason most descriptions sound confusing, inadequate, and/or incomprehensible. Often, the disseminating medium is a one-sided, like a video or a blog post, which doe…