Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Visualizing Attention, a Transformer's Heart [video]

3blue1brown.com

Post reply on HN
Print view View original

51–60 of 181 posts

Re: Visualizing Attention, a Transformer's Heart [video]

#51

Post by albertzeyer »

mjburgess wrote: post #45
seydor wrote: post #31

I have found the youtube videos by CodeEmporium to be simpler to follow https://www.youtube.com/watch?v=Nw_PJdmydZY Transformer is hard to describe with analogies, and TBF there is no good explanation why it works, so it may be better to just present the mechanism, "leaving the interpretation to the viewer". Also, it's simpler to describe dot products as vectors projecting on one another

The explanation is just that NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). Their weights are a model of this distribution. LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data. Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model…

You are more speaking about n-gram models here. NNs do far more than that.

Or if you just want to say that NNs are used as a statistical model here: Well, yea, but that doesn't really tell you anything. Everything can be a statistical model.

E.g., you could also say "this is exactly the way the human brain works", but it doesn't really tell you anything how it really works.

Re: Visualizing Attention, a Transformer's Heart [video]

#52

Post by Al-Khwarizmi »

bilsbie wrote: post #6

I finally understand this! Why did every other video make it so confusing!

Not sure if you mean it as rhetorical question but I think it's an interesting question. I think there are at least three factors why most people are confused about Transformers:

1. The standard terminology is "meh" at most. The word "attention" itself is just barely intuitive, "self-attention" is worse, and don't get me started about "key" and "value".

2. The key papers (Attention is All You Need, the BERT paper, etc.) are badly written. This is probably an unpopular opinion. But note that I'm not diminishing their merits. It's perfectly compatible to write a hugely impactful, transformative paper describing an amazing breakthrough, but just don't explain it very well. And that's exactly what happened, IMO.

3. The way in which these architectures were discovered was largely by throwing things at the wall and seeing what sticked. There is no reflection process that ended on a prediction that such an architecture would work well, which was then empirically verified. It's empirical all the way through. This means that we don't have a full understanding of why it works so well, all explanations are post hoc rationalizations (in fact, lately there is some work implying that other architectures may work equally well if tweaked enough). It's hard to explain something that you don't even fully understand.

Everyone who is trying to explain transformers has to overcome these three disadvantages... so most explanations are confusing.

Re: Visualizing Attention, a Transformer's Heart [video]

#53

Post by ffwd »

seydor wrote: post #48
mjburgess wrote: post #45

Earlier quoted context omitted.

The explanation is just that NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). Their weights are a model of this distribution. LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data. Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model…

People specifically would like to know what the attention calculations add to this learning of the distribution

Just speculating but I think attention enables differentiation of semantic concepts for a word or sentence within a particular context. Like for any total set of training data you have a lesser number of semantic concepts (like let's say you have 10000 words, then it might contain 2000 semantic concepts, and those concepts are defined by the sentence structure and surrounding words, which is why they have a particular meaning), and then attention allows to differentiate those different contexts at different levels (words/etc). Also the fact you can do this attention at runtime/inference means you can generate the context from the prompt, which enables the flexibility of variable prompt/variable output but you lose the precision of giving an exact prompt and getting an exact answer

Re: Visualizing Attention, a Transformer's Heart [video]

#54

Post by mjburgess »

albertzeyer wrote: post #51
mjburgess wrote: post #45

Earlier quoted context omitted.

The explanation is just that NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). Their weights are a model of this distribution. LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data. Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model…

You are more speaking about n-gram models here. NNs do far more than that. Or if you just want to say that NNs are used as a statistical model here: Well, yea, but that doesn't really tell you anything. Everything can be a statistical model. E.g., you could also say "this is exactly the way the human brain works", but it doesn't really tell you anything how it really works.

My description is true of any statistical learning algorithm.

The thing that people are looking to for answers, the NN itself, does not have them. That's like looking to Newton's compass to understand his general law of gravitation.

The reason that LLMs trained on the internet and every ebook has the structure of human communication is because the dataset has that structure. Why does the data have that structure? this requires science, there is no explanation "in the compass".

NNs are statistical models trained on data -- drawing analogies to animals is a mystification that causes people's ability to think clearly he to jump out the window. No one compares stock price models to the human brain; no banking regulator says, "well your volatility estimates were off because your machines had the wrong thoughts". This is pseudoscience.

Animals are not statistical learning algorithms, so the reason that's uninformative is because it's false. Animals are in direct causal contact with the world and uncover its structure through interventional action and counterfactual reasoning. The structure of animal bodies, and the general learning strategies are well-known, and having nothing to do with LLMs/NNs.

The reason that I know "The cup is in my hand" is not because P("The cup is in my hand"|HistoricalTexts) > P(not "The cup is in my hand"|HistoricalTexts)

Re: Visualizing Attention, a Transformer's Heart [video]

#55

Post by mjburgess »

nerdponx wrote: post #47
mjburgess wrote: post #45

Earlier quoted context omitted.

The explanation is just that NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). Their weights are a model of this distribution. LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data. Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model…

I think you're downplaying the importance of the attention/transformer architecture here. If it was "just" a matter of throwing compute at probabilities, then we wouldn't need any special architecture at all. P(next_word|previous_words) is ridiculously hard to estimate in a way that is actually useful. Remember how bad text generation used to be before GPT? There is innovation in discovering an architecture that make…

Yes, it's really hard -- the innovation is aligning the really basic dot-product similarity mechanism to hardware. You can use basically any NN structure to do the same task, the issue is that they're untrainable because they arent parallizable.

There is no innovation here in the sense of a brand new algorithm for modelling conditional probabilities -- the innovation is in adapting the algorithm for GPU training on text/etc.

Re: Visualizing Attention, a Transformer's Heart [video]

#56

Post by WithinReason »

Solvency wrote: post #8
bilsbie wrote: post #6

I finally understand this! Why did every other video make it so confusing!

Because: 1. good communication requires an intelligence that most people sadly lack 2. because the type of people who are smart enough to invent transformers have zero incentive to make them easily understandable. most documents are written by authors subconsciously desperate to mentally flex on their peers.

2. It's not malice. The longer you have understood something the harder it is to explain it, since you already forgot what it was like to not understand it.

Re: Visualizing Attention, a Transformer's Heart [video]

#57

Post by mjburgess »

forrestthewoods wrote: post #49
mjburgess wrote: post #45

Earlier quoted context omitted.

The explanation is just that NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). Their weights are a model of this distribution. LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data. Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model…

I find this take super weak sauce and shallow. This recent $10,000 challenge is super super interesting imho. https://twitter.com/VictorTaelin/status/1778100581837480178 State of the art models are doing more than “just” predicting the probability of the next symbol.

You underestimate the properties of the sequential-conditional structure of human communication.

Consider how a clever 6yo could fake being a physicist with access to a library of physics textbooks and a shredder. All the work is done for them. You'd need to be a physicist to spot them faking it.

Of course, LLMs are in a much better position than having shredded physics textbooks -- they have shreddings of all books. So you actually have to try to expose this process, rather than just gullibly prompt using confirmation bias. It's trivial to show they work this way, both formally and practically.

The issue is, practically, gullible people aren't trying.

Re: Visualizing Attention, a Transformer's Heart [video]

#58

Post by cmplxconjugate »

Al-Khwarizmi wrote: post #52
bilsbie wrote: post #6

I finally understand this! Why did every other video make it so confusing!

Not sure if you mean it as rhetorical question but I think it's an interesting question. I think there are at least three factors why most people are confused about Transformers: 1. The standard terminology is "meh" at most. The word "attention" itself is just barely intuitive, "self-attention" is worse, and don't get me started about "key" and "value". 2. The key papers (Attention is All You Need, the BERT paper, et…

>This is probably an unpopular opinion.

I wouldn't say so. Historically it's quite common. Maxwell's EM papers used such convoluted notation it it quite difficult to read. It wasn't until they were reformulated in vector calculus that they became infinitely more digestible.

I think though your third point is the most important; right now people are focused on results.

Re: Visualizing Attention, a Transformer's Heart [video]

#59

Post by rcarmo »

nerdponx wrote: post #44
seydor wrote: post #31

I have found the youtube videos by CodeEmporium to be simpler to follow https://www.youtube.com/watch?v=Nw_PJdmydZY Transformer is hard to describe with analogies, and TBF there is no good explanation why it works, so it may be better to just present the mechanism, "leaving the interpretation to the viewer". Also, it's simpler to describe dot products as vectors projecting on one another

> TBF there is no good explanation why it works My mental justification for attention has always been that the output of the transformer is a sequence of new token vectors such that each individual output token vector incorporates contextual information from the surrounding input token vectors. I know it's incomplete, but it's better than nothing at all.

You're effectively steering the predictions based on adjacent vectors (and precursors from the prompt). That mental model works fine.

Re: Visualizing Attention, a Transformer's Heart [video]

#60

Post by [deleted] »

nerdponx wrote: post #46
justanotherjoe wrote: post #43

It seems he brushes over the positional encoding, which for me was the most puzzling part of transformers. The way I understood it, positional encoding is much like dates. Just like dates, there are repeating minutes, hours, days, months...etc. Each of these values has shorter 'wavelength' than the next. The values are then used to identify the position of each tokens. Like, 'oh, im seeing january 5th tokens. I'm jan…

That's one way to think about it. It's clever way to encode "position in sequence" as some kind of smooth signal that can be added to each input vector. You might appreciate this detailed explanation: https://towardsdatascience.com/master-positional-encoding-pa... Incidentally, you can encode dates (e.g. day of week) in a model as sin(day of week) and cos(day of week) to ensure that "day 7" is mathematically adjacent…

[deleted]
Post reply on HN