Understand how transformers work by demystifying the math behind them

Transformers can be considered a kind of neural network.

It’s mainly fancy math. With tools like PyTorch or tensorflow, you use python to describe a graph of computations which gets compiled down into optimized instructions.

There are some examples of people making transformers and other NN architectures in about 100 lines of code. I’d google for those to see what these things look like in code.

The training loop, data, and resulting weights are where the magic is.

The code is disappointingly simple.

Just math, and not even that fancy.

Let's say you want to predict if you'll pass an exam based on how many hours you studied (x1) and how many exercises you did (x2). A neuron will learn a weight for each variable (w1 and w2). If the model learns w1=0.5 and w2=1, the model will provide more importance to the # of exercises.

So if you study for 10 hours and only do 2 exercises, the model will do x1w1 + x2w2=10x0.5 + 2x1 = 7. The neuron then outputs that. This is a bit (but not much) simplified - we also have a bias term and an activation to process the output.

Congrats! We built our first neuron together! Have thousands of these neurons in connected layers, and you suddenly have a deep neural network. Have billions or trillions of them, you have an LLM :)

Anyone telling you it does is a fraud.

  > The code is disappointingly simple.

I absolutely adore this sentence, it made me laugh to imagine coders or other folks looking at the code and thinking "That's it?!? But that's simple!"

Although it feels a little similar to some of the basic reactions that go to make up DNA: start with simple units that work together to form something much more complex.

(apologies for poor metaphors, I'm still trying to grasp some of the concepts involved with this)

It’s a renormalization process. It can be modelled as a convolution in a Hopf algebra.