Definitely more than meets the eye.
Understand how transformers work by demystifying the math behind them
91–100 of 139 posts
Re: Understand how transformers work by demystifying the math behind them
#92The whole "mystery" of transformer is that instead of a linear sequence of static weights times values in each layer, you now have 3 different matrices that are obtained from the same input through multiplication of learned weights, and then you just multiply the matrices together. I.e more parallelism which works out nice, but very restrictive since the attention formula is static. We arent going to see more progres…
How can gradient descent work on compute graphs when the space of compute graphs is discrete?
Perhaps in a way similar to this paper: https://arxiv.org/abs/1806.09055
Re: Understand how transformers work by demystifying the math behind them
#93Earlier quoted context omitted.
> We arent going to see more progress until we have a way to generalize the compute graph as a learnable parameter That's a bold statement since a ton of progress has been made without learning the compute graph.
A ton of progress can be made climbing a tree, but if your goal is reaching the moon it becomes clear pretty quickly that climbing taller trees will never get you there.
With enough thrust, even p̵i̵g̵s̵ trees can fly.
Re: Understand how transformers work by demystifying the math behind them
#94> Uh oh! We’re getting NaNs! It seems our values are too high, and when being passed to the next encoder, they end up being too high and exploding! This is called gradient explosion. As far as I understand this is wrong. You're not computing gradients at any point, so this is no gradient explosion. I believe the problem is with the implementation of softmax, here [0] you have an explanation of how to implement a nume…
Yes, you're correct. I tried to connect a common training problem (gradient explosion and vanishing gradient) with the issue of softmax being sensitive to large values. I agree it's misleading/inaccurate, so will rewrite that part.
That said, the whole neural network will be sensible to large values, so it won't be fixed by a numerically stable softmax. The normalization is a key aspect for the network to work.
Re: Understand how transformers work by demystifying the math behind them
#95Re: Understand how transformers work by demystifying the math behind them
#96Earlier quoted context omitted.
We have made progress in efficiency, not functionality. Instead of searching google or stack overflow or any particular documentation, we just go to Chatgpt. Information compression is cool, but I want actual AI.
Fascinating. What’s “actual AI”?
It’s whatever computers can’t do, dummy! :P
Re: Understand how transformers work by demystifying the math behind them
#97Not completely related. Does anyone know where I can find articles / papers that discuss why transformers, while acting as merely "next token predictor" can handle questions with: 1. Unknown words (or subwords/tokens) that are not seen in the training dataset. Example: Create a table with "sdsfs_ff", "fsdf_value" as columns in pandas. 2. Create examples(unseen in training dataset) and tell the LLM to provide similar…
It’s not reproducing exact strings in the training data but patterns and patterns of patterns. Next token prediction is more intelligent than it sounds
Reminds me of that person who asked chatgpt to make its own language with vocab and grammar rules and translate back and forth, it blew my mind
Re: Understand how transformers work by demystifying the math behind them
#98Earlier quoted context omitted.
Here [1] are some "frauds" from Stanford University, Oxford University and University College London telling you exactly that. From their abstract: ``One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial represent…
Making the claim that transformers are a good candidate model for certain neural pathways is a pretty different claim than saying the brain is literally using transformers.
In an alternative timeline, humans were being hooked up and used as GPUs in The Matrix
Re: Understand how transformers work by demystifying the math behind them
#99The whole "mystery" of transformer is that instead of a linear sequence of static weights times values in each layer, you now have 3 different matrices that are obtained from the same input through multiplication of learned weights, and then you just multiply the matrices together. I.e more parallelism which works out nice, but very restrictive since the attention formula is static. We arent going to see more progres…
can you explain exactly what you mean by this? I understand what a compute graph is, but I'm not getting the idea of making a learnable parameter.
Re: Understand how transformers work by demystifying the math behind them
#100The whole "mystery" of transformer is that instead of a linear sequence of static weights times values in each layer, you now have 3 different matrices that are obtained from the same input through multiplication of learned weights, and then you just multiply the matrices together. I.e more parallelism which works out nice, but very restrictive since the attention formula is static. We arent going to see more progres…
can you explain exactly what you mean by this? I understand what a compute graph is, but I'm not getting the idea of making a learnable parameter.
nevermind, after looking at my own question 3-4 times it clicked.