Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Visualizing Attention, a Transformer's Heart [video]

3blue1brown.com

Post reply on HN
Print view View original

71–80 of 181 posts

Re: Visualizing Attention, a Transformer's Heart [video]

#71

Post by mjburgess »

bruce343434 wrote: post #61
mjburgess wrote: post #55

Earlier quoted context omitted.

Yes, it's really hard -- the innovation is aligning the really basic dot-product similarity mechanism to hardware. You can use basically any NN structure to do the same task, the issue is that they're untrainable because they arent parallizable. There is no innovation here in the sense of a brand new algorithm for modelling conditional probabilities -- the innovation is in adapting the algorithm for GPU training on t…

I don't know why you seem to have such a bone to pick with transformers but imo it's still interesting to learn about it, and reading your dismissively toned drivel of "just" and "simply" makes me tired. You're barking up the wrong tree man, what are you on about.

No issue with transformers -- the entire field of statistical learning, decision trees to NNs, do the same thing... there's no mystery here. No person with any formal training in mathematical finance, applied statistics, hard experimental sciences on complex domains... etc. would be taken in here.

I'm trying my best to inform people who are interested in being informed, against an entire media ecosystem being played like a puppet-on-a-string by ad companies. The strategy of these companies is to exploit how easy is it to strap anthropomorphic interfaces over models of word frequencies and have everyone lose their minds.

Present the same models as a statistical dashboard, and few would be so adamant that their sci-fi fantasy is the reality.

Re: Visualizing Attention, a Transformer's Heart [video]

#72

Post by bjornsing »

This was the best explanation I’ve seen. I think it comes down to essentially two aspects: 1) he doesn’t try to hide complexity and 2) he explains what he thinks is the purpose of each computation. This really reduces the room for ambiguity that ruins so many other attempts to explain transformers.

Re: Visualizing Attention, a Transformer's Heart [video]

#73

Post by mjburgess »

IanCal wrote: post #68
mjburgess wrote: post #45

Earlier quoted context omitted.

The explanation is just that NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). Their weights are a model of this distribution. LLMs are a hardware innovation: they make it possible for GPUs to compute this at scale across TBs of data. Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent word in the dataset; and the NN is a model…

This is wrong, or at least a simplification to the point of removing any value. > NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). They are trained to maximise this, yes. > Their weights are a model of this distribution. That doesn't really follow, but let's leave that. > Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent wor…

Any NN "trained on" data sampled from an abstract complete outcome space (eg., a game with formal rules; mathematical sequences, etc) can often represent that space completely. It comes down to whether you can form conditional probability models of the rules, and that's usually possible because that's what abstract rules are.

> I have one kjsdhlisrnj and I add another kjsdhlisrnj, tell me how many kjsdhlisrnj I now have.

1. P(number-word|tell me how many...) > P(other-kinds-of-words|tell me how many...)

2. P(two|I have one ... I add another ...) > P(one|...) > P(three|...) > others

This is trivial.

Re: Visualizing Attention, a Transformer's Heart [video]

#74

Post by IanCal »

mjburgess wrote: post #73
IanCal wrote: post #68

Earlier quoted context omitted.

This is wrong, or at least a simplification to the point of removing any value. > NNs are a stat fitting alg learning a conditional probability distribution, P(next_word|previous_words). They are trained to maximise this, yes. > Their weights are a model of this distribution. That doesn't really follow, but let's leave that. > Why does, 'mat' follow from 'the cat sat on the ...' because 'mat' is the most frequent wor…

Any NN "trained on" data sampled from an abstract complete outcome space (eg., a game with formal rules; mathematical sequences, etc) can often represent that space completely. It comes down to whether you can form conditional probability models of the rules, and that's usually possible because that's what abstract rules are. > I have one kjsdhlisrnj and I add another kjsdhlisrnj, tell me how many kjsdhlisrnj I now h…

Right, learning more abstract rules about how things work is the goal and where the value comes in. Not all algorithms are able to do this, even if they can do what you describe in your first comment.

That's why they're interesting, othellogpt is interesting because it builds a world model.

Re: Visualizing Attention, a Transformer's Heart [video]

#75

Post by mjburgess »

vineyardmike wrote: post #70
mjburgess wrote: post #54

Earlier quoted context omitted.

My description is true of any statistical learning algorithm. The thing that people are looking to for answers, the NN itself, does not have them. That's like looking to Newton's compass to understand his general law of gravitation. The reason that LLMs trained on the internet and every ebook has the structure of human communication is because the dataset has that structure. Why does the data have that structure? thi…

> The reason that I know "The cup is in my hand" is not because P("The cup is in my hand"|HistoricalTexts) > P(not "The cup is in my hand"|HistoricalTexts) I mostly agree with your points, but I still disagree with this premise. Humans (and other animals) absolutely are statistical reasoning machines. They're just advanced ones which can process more than "text" - they're multi-modal. As a super dumb-simple set of ex…

I mean something specific by "statistics": modelling frequency associations in static ensembles of data.

Having a body which changes over time that interacts with a world that changes over time makes animal learning not statistical (call it, say, experimental). That animals fall into skinner-box irrational behaviour can be modelled as a kind of statistical learning, but it actually isnt.

It's a failure of ecological salience mechanisms in regulating the "experimental learning" that animals engage in. Eg., with the cargo cults the reason they adopted that view was because their society had a "big man" value system based on material acquisition and western waring powers seemed Very Big and so were humiliating. In order to retain their status they adopted (apparently irrational) theories of how the world worked (gods, etc).

From the outside this process might seem statistical, but it's the opposite. Their value system made material wealth have a different causal salience which was useful in their original ecology (a small island with small resources), but it went haywire when faced with the whole world.

Eventually these mechanisms update with this new information, or the tribe dies off -- but what's going wrong here is that the very very non-statistical learning ends up describable that way.

This is indeed, why we should be very concerned about people skinner-boxing themsleves with LLMs

Re: Visualizing Attention, a Transformer's Heart [video]

#76

Post by Xcelerate »

As someone with a background in quantum chemistry and some types of machine learning (but not neural networks so much) it was a bit striking while watching this video to see the parallels between the transformer model and quantum mechanics.

In quantum mechanics, the state of your entire physical system is encoded as a very high dimensional normalized vector (i.e., a ray in a Hilbert space). The evolution of this vector through time is given by the time-translation operator for the system, which can loosely be thought of as a unitary matrix U (i.e., a probability preserving linear transformation) equal to exp(-iHt), where H is the Hamiltonian matrix of the system that captures its “energy dynamics”.

From the video, the author states that the prediction of the next token in the sequence is determined by computing the next context-aware embedding vector from the last context-aware embedding vector alone. Our prediction is therefore the result of a linear state function applied to a high dimensional vector. This seems a lot to me like we have produced a Hamiltonian of our overall system (generated offline via the training data), then we reparameterize our particular subsystem (the context window) to put it into an appropriate basis congruent with the Hamiltonian of the system, then we apply a one step time translation, and finally transform the resulting vector back into its original basis.

IDK, when your background involves research in a certain field, every problem looks like a nail for that particular hammer. Does anyone else see parallels here or is this a bit of a stretch?

Re: Visualizing Attention, a Transformer's Heart [video]

#77

Post by spacecadet »

CamperBob2 wrote: post #21
spacecadet wrote: post #17

Earlier quoted context omitted.

Nah someone down voted it. And yes, it looks like that + 20 others that are animated.

Downvotes == empty boats. If "Empty Boat parable" doesn't ring a bell, Google it...

anger is a gift

Re: Visualizing Attention, a Transformer's Heart [video]

#78

Post by bugthe0ry »

rayval wrote: post #42

Here's a compelling visualization of the functioning of an LLM when processing a simple request: https://bbycroft.net/llm This complements the detailed description provided by 3blue1brown

When visualised this way, the scale of GPT-3 is insane. I can't imagine what 4 would like here.

Re: Visualizing Attention, a Transformer's Heart [video]

#79

Post by mjburgess »

IanCal wrote: post #74
mjburgess wrote: post #73

Earlier quoted context omitted.

Any NN "trained on" data sampled from an abstract complete outcome space (eg., a game with formal rules; mathematical sequences, etc) can often represent that space completely. It comes down to whether you can form conditional probability models of the rules, and that's usually possible because that's what abstract rules are. > I have one kjsdhlisrnj and I add another kjsdhlisrnj, tell me how many kjsdhlisrnj I now h…

Right, learning more abstract rules about how things work is the goal and where the value comes in. Not all algorithms are able to do this, even if they can do what you describe in your first comment. That's why they're interesting, othellogpt is interesting because it builds a world model.

It builds a model of a "world" whose structure is conditional probabilities, this is circular. It's like saying you can use a lego model to build a model of another lego model. All the papers which "show" NNs building "world" models arent using any world. It's lego modelling lego.

The lack of a world model only matters when the data NNs are trained on aren't valid measures of the world that data is taken to model. All the moves of a chess game are a complete model of chess. All the books ever written aren't a model of, well, anything -- the structure of the universe isnt the structure of text tokens.

The only reason all statistical algorithms, including NNs, appear to model the actual world is because patterns in data give this appearance: P(The Sun is Hot) > P(The Sun is Cold) -- there is no model of the sun here.

The reason P("The Sun is Hot") seems to model the sun, is because we can read the english words "sun" and "hot" -- it is we who think the machine which generates this text does so semantically.. but the people who wrote that phrase in the dataset did so; the machine is just generating "hot" because of that dataset.

Re: Visualizing Attention, a Transformer's Heart [video]

#80

Post by ffwd »

ffwd wrote: post #53
seydor wrote: post #48

Earlier quoted context omitted.

People specifically would like to know what the attention calculations add to this learning of the distribution

Just speculating but I think attention enables differentiation of semantic concepts for a word or sentence within a particular context. Like for any total set of training data you have a lesser number of semantic concepts (like let's say you have 10000 words, then it might contain 2000 semantic concepts, and those concepts are defined by the sentence structure and surrounding words, which is why they have a particula…

I'm not one to whine about downvotes but I just have to say, it's a bad feeling when I can't even respond to the negative feedback because there is no accompanying comment. Did I misinterpret something? Did you? Who will ever know when there is no information. :L
Post reply on HN