Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

news.ycombinator.com

Post reply on HN
Print view View original

191–200 of 240 posts

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#191

Post by kylewatson »

nafey wrote: post #184
dominickramer wrote: post #172

Suppose someone asked you to complete the sentence: “After I woke up and made breakfast, I drank a glass of …” In America one might say the most likely next words are “orange juice”, or “apple juice” but not “sports car” which has nothing to do with the sentence. Ultimately this is what language models do, given a sequence of data (in this case words) predict the most likely next word(s). For attention, when you read…

When one says "attention is all you need" the implication is that some believe that you need something more than just attention. What is that something which has been demonstrated as unneeded? Is it a theory of how language works?

I think this will help: https://medium.datadriveninvestor.com/attention-in-rnns-321f...

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#193

Post by golly_ned »

throwawaymaths wrote: post #63
saynay wrote: post #57

Earlier quoted context omitted.

I believe the other big thing is it allowed you to parallelize the training as well, so you could split those colossal training sets across many machines in a way you could not do with LSTMs.

I don't really believe this because in theory it's possible to split up training of arbitrary neural nets using decoupled neural interfaces (not that it's easy mind you) but at the point where you have openAI scale resources the bottleneck to achieving that is the difficulty of Python.

What’s a neural interface?

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#195

Post by brianjking »

soVeryTired wrote: post #148
QuantumGood wrote: post #147

Earlier quoted context omitted.

In the context of natural language processing, the attention mechanism used in Transformer models and the process of converting tokens to vectors and calculating cosine similarity have similarities but serve different purposes. When you convert words (tokens) into vectors and calculate cosine similarity, you're typically doing what's called "word embedding". This process captures the semantic meaning of words in a hi…

Just a guess: is this answer GPT output?

Probably, it is a good explanation, though.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#196

Post by Too »

benjismith wrote: post #90

Okay, here's my attempt! First, we take a sequence of words and represent it as a grid of numbers: each column of the grid is a separate word, and each row of the grid is a measurement of some property of that word. Words with similar meanings are likely to have similar numerical values on a row-by-row basis. (During the training process, we create a dictionary of all possible words, with a column of numbers for each…

If it only generates one word at a time and then repeat the process again, how does it know when to stop?

It feels like this method would create endless ramblings. But we all know you can ask Chatgpt to “summarize in one sentence” and it pulls it off. When speaking yourself you sort of have to think how to finish a sentence before you start it, to explain something cohesively, surely there must be something similar in the AI?

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#197

Post by burnished »

brianjking wrote: post #194

How many responses here are generated entirely or aided by the technology they are explaining? That is the real question.

Any sequence of on topic responses that are not phrased as rebuttals or disagreements are almost assuredly generated.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#198

Post by dang »

QuantumGood wrote: post #99

The "Attention is All You Need" paper introduced a new way for AI to read and understand language, much like how we might read a comic book. As you read each panel of a comic book, you don't just look at the words in the speech bubbles, but you also pay attention to who's talking, what they're doing, and what happened in the previous panels. You might pay more attention to some parts than others. This is sort of like…

Please don't post generated comments into HN threads. HN threads are for human discussion and we ban accounts that violate this.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#199

Post by teleforce »

I strongly agree with Steve on the potential of Nobel prize for the discovery and I want to add that the team probably going to win it within 10 years. For reference, Jack Kilby won his shared Nobel prize on integrated circuit (IC) that transforms the electronics and computer industry after more than 40 years of the original invention.

Here is a very informative video and blog by Google Cloud Tech team explaining the game-changing nature of the self-attention to understand the contexts of the words being used in any sentences as proposed by the paper [1],[2].

[1] Transformers, explained: Understand the model behind GPT, BERT, and T5:

https://youtu.be/SZorAJ4I-sA

[2] Corresponding blog post:

https://daleonai.com/transformers-explained

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#200

Post by joshhart »

Too wrote: post #196
benjismith wrote: post #90

Okay, here's my attempt! First, we take a sequence of words and represent it as a grid of numbers: each column of the grid is a separate word, and each row of the grid is a measurement of some property of that word. Words with similar meanings are likely to have similar numerical values on a row-by-row basis. (During the training process, we create a dictionary of all possible words, with a column of numbers for each…

If it only generates one word at a time and then repeat the process again, how does it know when to stop? It feels like this method would create endless ramblings. But we all know you can ask Chatgpt to “summarize in one sentence” and it pulls it off. When speaking yourself you sort of have to think how to finish a sentence before you start it, to explain something cohesively, surely there must be something similar i…

One of the “words” is a stop token, which represents the ending of text. So you can say the thing that maximizes coherence is to stop right then.
Post reply on HN