211–220 of 240 posts
Well here is my (a bit cynical) take on it. In the beginning, there was the matrix multiply. A simple neural network is a chain of matrix multiplies. Let's say you have your data A1 and weights W1 in a matrix. You produce A2 as A1xW1. Then you produce A3 as A2xW2, and so on. There are other operations in there like non-linearities (so that you can actually learn something interesting) and fancy batch norms, but let's…
Earlier quoted context omitted.
I don't really believe this because in theory it's possible to split up training of arbitrary neural nets using decoupled neural interfaces (not that it's easy mind you) but at the point where you have openAI scale resources the bottleneck to achieving that is the difficulty of Python.
What’s a neural interface?
Earlier quoted context omitted.
When you train a transformer, you're training what the next expected token is. You can train all positions of the sequence each in parallel rather than having to sequentially build up the memory state as you generate the sequence with an LSTM. Mind you the inference portion of a transformer is still sequentially bottlenecked since you don't know what the output sequence is supposed to be.
If you train in parallel, how do you combine the weights generated from different trainings in parallel?
Okay, here's my attempt! First, we take a sequence of words and represent it as a grid of numbers: each column of the grid is a separate word, and each row of the grid is a measurement of some property of that word. Words with similar meanings are likely to have similar numerical values on a row-by-row basis. (During the training process, we create a dictionary of all possible words, with a column of numbers for each…
Andrej Karpathy has a great course on this that is accessible to anyone who knows calculus and basic coding: https://karpathy.ai/zero-to-hero.html
The best way to understand transformers is to take Andrej’s Karpathy course on youtube. With a keyboard and a lot of focus time.
There is a great discord community attached too which makes a big difference.
What is missing from this and another course I did and is very hard to find is multivariate calculus on linear algebra. I feel motivated to create a resource on it because its pretty hard. For example how to differentiate matrix operations where broadcasting has been involved. Not just the how but really grokking it into working memory.
Earlier quoted context omitted.
When you train a transformer, you're training what the next expected token is. You can train all positions of the sequence each in parallel rather than having to sequentially build up the memory state as you generate the sequence with an LSTM. Mind you the inference portion of a transformer is still sequentially bottlenecked since you don't know what the output sequence is supposed to be.
If you train in parallel, how do you combine the weights generated from different trainings in parallel?
Okay, here's my attempt! First, we take a sequence of words and represent it as a grid of numbers: each column of the grid is a separate word, and each row of the grid is a measurement of some property of that word. Words with similar meanings are likely to have similar numerical values on a row-by-row basis. (During the training process, we create a dictionary of all possible words, with a column of numbers for each…
I have a very dumb question, I'll just throw it here: I understand word embeddings and tokenisation- and the value of each; but how can the two work together? Are embeddings calculated for tokens, and in that case, how useful are they, given that each token is just a fragment of a word, often with little or no semantic meaning?
https://peterbloem.nl/blog/transformers
https://e2eml.school/transformers.html
I would also add Luis Serrano's article here: https://txt.cohere.com/what-are-transformer-models/ (HN discussion: https://news.ycombinator.com/item?id=35576918).
Looking back at The Illustrated Transformer, when I introduce people to the topic now, I find I can hide some complexity by omitting the encoder-decoder architecture and focusing only on one. Decoders are great because now a lot of people come to Transformers having heard of GPT models (which are decoder only). So for me, my canonical intro to Transformers now only touches on a decoder model. You can see this narrative here: https://www.youtube.com/watch?v=MQnJZuBGmSQ
Pro tip: if you want technical info on research-related topics, don't ask HN. Tech bros can't handle telling themselves they don't know something, so everyone will give their "take" on the subject at hand.