Feedforward: y=Wx
Attention: y=W(x)x
W is Matrix, x & y Are vectors. In the second case, W is a function of the input.
Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
31–40 of 240 posts
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#32Who is Yegge and why should I care? All I found on google is that he wrote code for google and blogs.
I first heard about him after an internal post/rant of his about Google+ got leaked and went viral.
My impression is that he is a Kardashian - famous for being famous - in the software world.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#33[flagged]
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#34[flagged]
Maybe ELI5 was the wrong level for my ask. I still want to understand it from a technical perspective.
Maybe ELI am a software engineer but not in AI/ML?
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#35Check this explanation from Karpathy which provides really good intuition https://youtu.be/9uw3F6rndnA Basically tokens “talk” to each other and say this is what i have and this is what i look for.
Your summary of this is very good. It’s a really simple and intuitive way to express something very complex.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#36Transformers are about converting some input data (usually text) to numeric representations, then modifying those representations through several layers to generate a target representation. In LLMs, this means go from prompt to answer. I'll cover inference only, not training. I can't quite ELI5, but process is roughly: - Write a prompt - Convert each token in the prompt (roughly a word) into numbers. So "the" might m…
I don't think this description of attention is correct.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#37The Yannic kilcher review is quite good. https://youtu.be/iDulhoQ2pro I can't ELI5 but I can ELI-junior-dev. Tl;dw: Transformers work by basically being a differentiable lookup/hash table. First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame. Then there is an NxN matrix that is applied to your attention frame "…
One thing that might be worth pointing out is that the transformer architecture owes a great deal of its success to the fact that it can be implemented in a way that it can be massively parallelized in a very efficient manner.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#38Feedforward: y=Wx Attention: y=W(x)x W is Matrix, x & y Are vectors. In the second case, W is a function of the input.
You must be from a planet with very long years!
There is no way I can even begin to digest what you have said in your comment.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#39[flagged]
Absolute BS. Transformers were literally introduced by that paper.
Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?
#40The Yannic kilcher review is quite good. https://youtu.be/iDulhoQ2pro I can't ELI5 but I can ELI-junior-dev. Tl;dw: Transformers work by basically being a differentiable lookup/hash table. First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame. Then there is an NxN matrix that is applied to your attention frame "…
One thing that might be worth pointing out is that the transformer architecture owes a great deal of its success to the fact that it can be implemented in a way that it can be massively parallelized in a very efficient manner.
Compared to rnns... maybe? The big nxn is really a killer.
I don't know how to judge parallelizability of different DNN models, you're comparing apples to oranges