Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 240 posts

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#71

Post by throwawaymaths »

meowkit wrote: post #62
kenjackson wrote: post #54

Earlier quoted context omitted.

What does it mean for a lookup/hash table to be differentiable?

Going to go out on a limb and say they are probably referring to the gradient calculus required for updating the model. https://en.wikipedia.org/wiki/Differentiable_programming See automatic differentiation.

Correct, but note that if you subject a standard hash table algo to AD it won't magically become a transformer. (Hashes in the "normal construction" are discrete functions and thus aren't really continuous or differentiable, neither are lookup tables)

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#72

Post by contravariant »

kenjackson wrote: post #54
throwawaymaths wrote: post #8

The Yannic kilcher review is quite good. https://youtu.be/iDulhoQ2pro I can't ELI5 but I can ELI-junior-dev. Tl;dw: Transformers work by basically being a differentiable lookup/hash table. First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame. Then there is an NxN matrix that is applied to your attention frame "…

What does it mean for a lookup/hash table to be differentiable?

Well, a hash table is basically a function.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#73

Post by throwawaymaths »

taylorius wrote: post #68
kenjackson wrote: post #54

Earlier quoted context omitted.

What does it mean for a lookup/hash table to be differentiable?

Differentiable in this context, means that it can be learned by gradient descent, which uses derivatives to adjust a neural network model's parameters to minimise some error measure. As for how that applies to a hash function, I think the lookup gives some sort of weighting for each possible output, for each input, with the largest weightings corresponding to a "match". But tbh I'm not certain on that last part...

> As for how that applies to a hash function, I think the lookup gives some sort of weighting for each possible output, for each input, with the largest weightings corresponding to a "match"

Note that in general one "matches" more than one thing in which case (IIRC) the result will be a weighted linear combination of the query results.

I do think that this collapses to a datastructure equivalent in function to the "normal hash table" (but not in performance, lol) in the degenerate case where every input yields strictly a single result with zero activation in all other results, so it's not invalid to call it a differentiable lookup table.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#74

Post by jasonjmcghee »

tikkun wrote: post #44
factorymoo wrote: post #12

"Transformers" and "Attention is All You Need" refer to an important development in machine learning and artificial intelligence, particularly in the field of natural language processing (NLP). I'll try to explain them in a simple way. Think of a conversation you had with a friend. While they were talking, you were probably not just listening to the words they were saying right now, but also remembering what they sai…

Did you use GPT to write this? (Not a bad thing! It's a decent answer)

Immediately thought this was gpt as well.

Assuming prompt was "Explain Transformers and 'Attention is all you need' in a simple way"

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#75

Post by theGnuMe »

kenjackson wrote: post #54
throwawaymaths wrote: post #8

The Yannic kilcher review is quite good. https://youtu.be/iDulhoQ2pro I can't ELI5 but I can ELI-junior-dev. Tl;dw: Transformers work by basically being a differentiable lookup/hash table. First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame. Then there is an NxN matrix that is applied to your attention frame "…

What does it mean for a lookup/hash table to be differentiable?

The lookup/hashtable can be viewed as a matrix which is accessed by multiplying it with a vector.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#76

Post by visarga »

kenjackson wrote: post #54
throwawaymaths wrote: post #8

The Yannic kilcher review is quite good. https://youtu.be/iDulhoQ2pro I can't ELI5 but I can ELI-junior-dev. Tl;dw: Transformers work by basically being a differentiable lookup/hash table. First your input is tokenized and (N) tokens (this constitutes the attention frame) are encoded both based on token identity and position in the attention frame. Then there is an NxN matrix that is applied to your attention frame "…

What does it mean for a lookup/hash table to be differentiable?

It's not really doing hashing, which is random, but instead it makes meaningful connections, like connecting an adjective to its noun, or a pronoun to a previous reference of the name. That is why it is called "Attention is all you need" and not "Hashing is all you need". Attention includes some meaning, it is attentive to some aspect or another.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#77

Post by jvm___ »

So, if you got a time machine back to the year 2000 and knew how to code Transformers - could you do it? Your CPU/cloud/parallel processing hardware would be limited, but as I understand it there's nothing revolutionary being done here, just a lot of matrix math that produces results (and we're still fully understanding why that all happens).

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#78

Post by visarga »

weinzierl wrote: post #59
kenjackson wrote: post #54

Earlier quoted context omitted.

What does it mean for a lookup/hash table to be differentiable?

I wanted to ask the same and especially I've always been wondering: How is the meaning of aforementioned 'differentiable' related to the same term in math?

The neural net is just a math function, continuous even, fully differentiable in all input points. In order to "learn" anything we compute gradients towards the function parameters. They get "nudged" slightly towards a better response, and we do this billions of times. It's like carving a raw stone block into a complex scene. If you put your data into the system it flows towards the desired output because the right path has been engraved during training.

This explains a bit how neural nets work but from this to chatGPT is another whole leap. You'd have to assign some of the merits of the AI to the training data itself, it's not just the algorithm for learning, but what is being learned that matters. The neural net is the same, but using 1T tokens of text is making it smart. What's so magic about this data that it can turn a random init into a language programmable system? And the same language information make a baby into a modern human, instead of just another animal.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#79

Post by raydiatian »

edgyquant wrote: post #46
raydiatian wrote: post #18

It’s not really something you need to understand unless you’re an ML researcher. I guess the ELI5 (with a BUNCH of details left out) is “Transformers: what if you didn’t have to process sentences as a sequence of words, but rather as a picture of words.”

>t’s not really something you need to understand unless you’re an ML researcher What kind of an answer is this? No it’s something you should understand if you are at all interested.

It’s like saying “I have no basis in differential equations or complex numbers or Newtonian mechanics or experimental findings on the wavelike nature of light but could somebody explain the Schrödinger equation derivation like I’m five?”

Like, there’s a reason we don’t explain things to five year olds.

Also I have a good ELI5 answer, so I don’t know what else you want dude lol so petty

Post reply on HN