This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.
Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
11–20 of 111 posts
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#12I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…
Step number 1 is already the first problem.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#13I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…
Extremely interested in your take on where language/reasoning competency ends and knowledge retrieval begins.
OpenAI stuff has succeeded in part because it can synthesize good bullshit* on a huge variety of topics. For many purposes this makes it as good as asking someone in the same room to look something up for you on Wikipedia.
But while vast general and somewhat special knowledge is very impressive, comprehension and reasoning ability can exist without it. We know from our own human experience that general knowledge is useful to have, but not the same thing as intelligence or wisdom. It seems rational to think that the size of model needed to get ChatGPT's adequate level coherence and rationality is much less than that required to also encode sufficient general knowledge to be informative on just about any topic, most of which are not language specific.
* in the Frankfurtian sense of 'information provided without regard to its correctness'
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#14I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…
On number 2, even if you are John Carmack you may have trouble getting the right people on the phone.
https://twitter.com/id_aa_carmack/status/1305967411749892098...
Anyone at Google Cloud out there? It seems I can't get my GPU quota raised to 40 x V100 as an independent researcher. I was told that setting up a website would help, but I would rather not. I can pay the bills...
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#15I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…
I was so confused by the saltiness until I saw the username. I'm sure you've earned it.
I got into deep learning because of your char-rnn posts a while ago -- it inspired me to do an undergrad thesis on the topic. I read arxiv papers after that and implemented things from the ground up until a startup liked my work and hired me in a neural network engineer position.
Fast forward a few years and I was enamoured with minGPT and it stuck with me. I wanted a CIFAR10 experimentation toolbench so I took my hand at my best swing at applying the minGPT treatment on the current best single-GPU Dawnbench entry, added a few tweaks and got https://github.com/tysam-code/hlb-CIFAR10. It currently (AFAIK) holds the world record for training to the 94% mark by a fair bit.
It's about 600 lines in a monolithic file, only requiring torch and torchvision, but it's my first project like this and I'd like to learn how to better minify codebases like this. It seems like the hardest part is knowing how to structure inheritance and abstraction, but I don't know if you had any good outside references/resources that you used or would recommend. If you have any feedback or help, I am open to receiving it, as I am very much a newbie at this particular art/science. It is quite a fun one, however (especially as it is a useful tool for my day-to-day work).
I'm also hoping to apply the same treatment to a small language model at some point by taking the Dawnbench approach -- picking a good target validation loss value or some reasonable metric, then optimize around that obsessively to build a good tiny reference model. I don't know if you'd know anyone that's interested in that kind of thing, but I feel like that would be a fun next step for me.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#16Karpathy's videos (and blogs) are excellent. I wonder how history will reflect on his time at Tesla, however.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#17Earlier quoted context omitted.
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…
Step number 1 is already the first problem.
Today to me is the equivalent of the phone phreaking days when people are just doing as much as they can and getting away with as much as they can for as long as they can, until the regulations come.
It will be an interesting time in the next few years as I think StabilityAI's guerilla marketing tactics have inadvertently by proxy also placed the ML dataset debate right in the laps of the larger consumer market.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#18Earlier quoted context omitted.
He's building the model from scratch, as the title suggests. He only trains a small model with 10M parameters on it, something that is feasible with a single GPU. In comparison, GPT-3 has 175B parameters. > wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training? It would most certainly be possible for another company to build something very similar (model…
Does the time to train the model increase linearly with the number of parameters, or exponentially? In other words, GPT-3 is 17,500X the number of parameters but does that mean you can train it in 17,500X the amount of time it takes to train the 10M param model?
I am not from the LLM world, but I believe it's mostly constrained by the standard multiprocessing limits -- communication and synchronization of multiple workers, some of whom operate over an exceedingly slow Ethernet interface.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#19This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.
Same, but one that doesn't assume python or an operating system.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#20This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.
Same, but one that doesn't assume python or an operating system.
Same, but I would honestly rather we get away from silicon or anything resembling what we call these "computers" these days. Things are too complicated and we need to truly get back to the basics: rocks on a hill.