Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 240 posts

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#151

Post by hoosieree »

tomp wrote: post #92
kenjackson wrote: post #54

Earlier quoted context omitted.

What does it mean for a lookup/hash table to be differentiable?

I'm not a ML expert but I know a bit about math. It's "differentiable" in the same way that e.g. the "jump function" ( Heaviside step function ) is differentiable (not as a function from real numbers to real numbers, but as a distribution ). It's derivative is the "point impulse function" ( Dirac delta function ), which, again, is a distribution , not a real function. Distributions are nicely defined in math, but can…

Thanks for this explanation. I couldn't wrap my mind around the "differentiable hash table" analogy, but "distribution of keys" -> "distribution of values" starts to click.

I'm not an ML expert either but I have taken graduate level courses and published papers with "machine learning" in the title, so I feel like I should be able to understand these things better. The field just moves so fast. It's a lot of work to keep up. Easy-to-digest explanations like this are underrated.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#153

Post by ggm »

the use of words like "attention" or "hallucinate" as 'terms of art' concerns me because while its actually fine and normal to re-purpose normal english discourse words to have highly specific meaning in a field, When you use them outside the cogniscenti, they carry meaning which is not strictly appropriate.

In this case, AGI proponents are using words which are highly loaded to mean "is a thinking, reasoning being" in some way.

I don't like it. I would prefer that they'd chosen words which were more neutral and not based on illusions of intelligence, or allusions to known intelligent behaviour.

"attention" is a thing, sure. But, if you use this word in formal session presenting to Congress, you're misleading them without conscious effort to believe you think "it's alive"

I don't like it. I think in hindsight calling the field AI was a huge mistake.

If you want something to hang on this, think about legal english and words like "real property" -do you really know what a solicitor or lawyer or barrister or judge means when they say that? or "without let or hindrance" -what does the word "let" mean there?

Within legal contexts, using the jargon is a given. misusing them outside the courtroom as a non-legal practitioner is a recipe for disaster. This is were "Sovereign Citizens" are playing: look how well that's going.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#154

Post by rshm »

UncleOxidant wrote: post #150
QuantumGood wrote: post #108

Earlier quoted context omitted.

Explaining it for a slightly older audience, a transformer is a type of artificial neural network designed for processing sequences, like sentences in a text. It's especially known for its use in natural language processing (NLP), which is the field of AI that deals with understanding and generating human language. The Transformer is unique because it uses a mechanism called "attention" to understand the relationship…

Someone who read the paper pointed out to me recently that there's an aspect to transformers/attention that uses the sin or cos function to determine which words to pay attention to or the spacing between them (I'm probably not expressing this correctly, so please correct me if I'm wrong). It seems really unintuitive that sin and/or cos would be a factor in human language - can you explain this?

Someone else can better explain this. Based on one of the video suggested in one of the replies here. sin and cos doesn’t have any inherent properties specific to language, they were chosen because a simple linear function was needed in that step of optimization. Any other function could fit the bill as well.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#155

Post by cwillu »

notfried wrote: post #120

Prompt: "ELI5 the Attention is All We Need paper." Imagine you have a box of toys. Some toys are more interesting to you than others, so you pay more attention to those toys and play with them more. The same thing happens in the "Attention is All You Need" paper, but instead of toys, we have words in a sentence. Before this paper, when computers tried to translate one language to another, they would look at one word,…

Prompt: “Why might people be hostile towards others posting simple copy-paste's of a chatgpt dialog in response to a question?”

Response:

“There could be several reasons why people might be hostile towards others who post simple copy-paste responses from ChatGPT in a dialog:

① Lack of originality: Copy-pasting a ChatGPT response without any personal input or effort can be seen as lazy or unoriginal. It shows a lack of engagement and genuine interaction with the question or conversation. People might value unique perspectives and insights, so when they see someone simply copying and pasting a response, they may perceive it as a low-quality contribution.

② […] ③ […] ④ […]

It's important to note that individual reactions may vary, and not everyone will respond with hostility in such situations. However, these reasons provide some insight into why certain individuals might have negative reactions towards simple copy-paste responses from ChatGPT.”

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#156

Post by sn41 »

QuantumGood wrote: post #108
QuantumGood wrote: post #99

The "Attention is All You Need" paper introduced a new way for AI to read and understand language, much like how we might read a comic book. As you read each panel of a comic book, you don't just look at the words in the speech bubbles, but you also pay attention to who's talking, what they're doing, and what happened in the previous panels. You might pay more attention to some parts than others. This is sort of like…

Explaining it for a slightly older audience, a transformer is a type of artificial neural network designed for processing sequences, like sentences in a text. It's especially known for its use in natural language processing (NLP), which is the field of AI that deals with understanding and generating human language. The Transformer is unique because it uses a mechanism called "attention" to understand the relationship…

complete newbie here: what is the intuition behind the conclusion that "cat" is highly related to "black" as opposed to, say, "mat"?

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#158

Post by martin82 »

Without animated visuals, I don't think any non-math/non-ML person can ever get a good understanding of transformers.

You will need to watch videos.

Watch this playlist and you will understand: https://youtube.com/playlist?list=PLaJCKi8Nk1hwaMUYxJMiM3jTB...

Then watch this and you will understand even more: https://youtu.be/g2BRIuln4uc

Finally, watch this playlist: https://youtube.com/playlist?list=PL86uXYUJ7999zE8u2-97i4KG_...

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#159

Post by antegamisou »

godelski wrote: post #42

> I have zero AI/ML knowledge This may make it difficult to explain and I already see many incorrect explanations here and even more lazy ones (why post the first Google result? You're just adding noise) > Steve Yegge on Medium thinks that the team behind Transformers deserves a Nobel First, Yegge needs to be able to tell me what Attention and Transformers are. More importantly, he needs to tell me who invented them.…

As usual, the most helpful answer is burried among desperate platitudes full of inaccuracies trying to pander to the absurd reddit-esque ELI5 notion you've dismantled.

Re: Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper?

#160

Post by aGHz »

sn41 wrote: post #156
QuantumGood wrote: post #108

Earlier quoted context omitted.

Explaining it for a slightly older audience, a transformer is a type of artificial neural network designed for processing sequences, like sentences in a text. It's especially known for its use in natural language processing (NLP), which is the field of AI that deals with understanding and generating human language. The Transformer is unique because it uses a mechanism called "attention" to understand the relationship…

complete newbie here: what is the intuition behind the conclusion that "cat" is highly related to "black" as opposed to, say, "mat"?

It is a lot harder to take the black out of the cat than it is to take the mat out from under it.
Post reply on HN