Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understand how transformers work by demystifying the math behind them

osanseviero.github.io

Post reply on HN
Print view View original

51–60 of 139 posts

Re: Understand how transformers work by demystifying the math behind them

#51

Post by kevindamm »

ActorNightly wrote: post #6

The whole "mystery" of transformer is that instead of a linear sequence of static weights times values in each layer, you now have 3 different matrices that are obtained from the same input through multiplication of learned weights, and then you just multiply the matrices together. I.e more parallelism which works out nice, but very restrictive since the attention formula is static. We arent going to see more progres…

hyperparameter tuning does already go some of the way towards learning the compute graph, though very constrained and with a lot more training required.

Re: Understand how transformers work by demystifying the math behind them

#52

Post by bsenftner »

Hard to understand when concepts are used without definition or introduction. The Encoder section just begins without any description of what it is or where is sets in an overall process. I grasp what the author is trying to do, but the post misses basic essay structures such as introducing ideas and explaining them before using them, rending the entire post confusing if one is not already a student and half understands the topic before reading.

Re: Understand how transformers work by demystifying the math behind them

#54

Post by kevindamm »

dogline wrote: post #40
dan-robertson wrote: post #31

Earlier quoted context omitted.

The number reuse is just the author being a bit lazy. You could estimate how similar these vectors are by seeing if they point in similar directions or by calculating the angle between them. Here they are about 60° apart and somewhat the same direction, but a lot of this is that the author didn’t want to put in any negative numbers in the example so vectors end up being a bit more similar than they would be really. T…

Thank you. I guess I need to back up. This is a vector, not just an identifier, and direction and angle seem important. I need to look up how the encoding is normally done, since this isn't obvious if you haven't worked in this domain before.

The encoding is typically learned, and if possible is part of the ANN so that it can be adjusted along with the other parameters.

A good place to start on that topic is the word2vec paper.

Re: Understand how transformers work by demystifying the math behind them

#55

Post by sroussey »

ActorNightly wrote: post #10
ex3ndr wrote: post #8

Earlier quoted context omitted.

This is basically this - it can learn ignore some paths, and amplify something more important, then you can just cut this paths without sensible loss of quality. The problem is that you are not going to win anything from this - non-matrix multiplication would be slower or the same.

The issue is that you are thinking of this in terms of information compression, which is what LLMs are. Im more concerned with an LLM having the ability to be trained to the point where a subset of the graph represents all the nand gates necessary for a cpu and ram, so when you ask it questions it can actually run code to compute them accurately instead of offering a statistical best guess, i.e decompression after lo…

Well, just remember that NAND gates are made of transistors themselves which are a statistical model of a sort… just designed to appear digital when combined to that NAND level.

This is why I am very interested in analog again—quantum stuff is statistical already, so why go from statistical (analog) to digital (huge drop off of performance, e.g. just look at basic addition in a ALU) and back to statistical. Very interested. Not sure if it will ever be worth it, but can’t rule it out.

Re: Understand how transformers work by demystifying the math behind them

#56

Post by manojlds »

nomel wrote: post #41
necroforest wrote: post #35

Earlier quoted context omitted.

> We arent going to see more progress until we have a way to generalize the compute graph as a learnable parameter That's a bold statement since a ton of progress has been made without learning the compute graph.

From my naive perspective, there seems to be a plateau, that everyone is converging on, somewhere between ChatGPT 3.5 and 4 level of performance, with some suspecting that the implementation of 4 might involve several expert models, which would already be extra sauce, external to the LLM. This, combined with the observation that generative models converge to the same output, given the same training data, regardless o…

We already have competitors to Transformers

https://arxiv.org/abs/2312.00752

Re: Understand how transformers work by demystifying the math behind them

#57

Post by tnecniv »

knightoffaith wrote: post #47
ActorNightly wrote: post #6

The whole "mystery" of transformer is that instead of a linear sequence of static weights times values in each layer, you now have 3 different matrices that are obtained from the same input through multiplication of learned weights, and then you just multiply the matrices together. I.e more parallelism which works out nice, but very restrictive since the attention formula is static. We arent going to see more progres…

How can gradient descent work on compute graphs when the space of compute graphs is discrete?

It can’t. There’s no gradient since it’s not a sufficiently nice space for them. You can use gradient free methods but I’d be shocked if there was an efficient enough way to do that

Re: Understand how transformers work by demystifying the math behind them

#58

Post by patcon »

willy_k wrote: post #39
brcmthrowaway wrote: post #11

Does the human brain use transformers?

I’m assuming you are asking if the brain uses transformer-like structures or otherwise exhibits similar behavior. I don’t know, but it does share some processes with simpler ML ideas, and I’d be very interested to see if it uses anything resembling a transformer.

Forward-forward algorithm is more like the brain. As I understand, backpropagation transformers require storing data, doing calculations on that aggregate, and sending it back through, which no neural structures can do anything like.

https://medium.com/@Mosbeh_Barhoumi/forward-forward-algorith...

Re: Understand how transformers work by demystifying the math behind them

#59

Post by __loam »

utopcell wrote: post #44
__loam wrote: post #25

Earlier quoted context omitted.

Anyone telling you it does is a fraud.

Here [1] are some "frauds" from Stanford University, Oxford University and University College London telling you exactly that. From their abstract: ``One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial represent…

Making the claim that transformers are a good candidate model for certain neural pathways is a pretty different claim than saying the brain is literally using transformers.
Post reply on HN