Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
1–10 of 111 posts
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#2Andrej’s entire series is by far one of the most useful resources I’ve found. Even as an instructor of these topics myself I learn something new about the material and about how to teach it!
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#3Fantastic material! By the way, one of the simplest explanation of the difference between BetchNorm and LayerNorm
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#4He has a website[0] for these videos
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#5I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training?
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#6I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
rough steps:
1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $.
2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts.
3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of MegatronLM+DeepSpeed. go through training hell, learn all about every possible NCCL error message, see the OPT logbook as good reference: https://github.com/facebookresearch/metaseq/blob/main/projec...
4. follow the 3-step recipe of https://openai.com/blog/chatgpt/ to finetune the model to be an actual assistant instead of just "document completor", which otherwise happily e.g. responds to questions with more questions. Also e.g. see OPT-IML https://arxiv.org/abs/2212.12017 , or BLOOMZ https://arxiv.org/abs/2211.01786 to get a sense of the work involved here.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#7I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
Yeah, he's explaining how you would create the base model, which is actually one of the more straightforward parts given that they've published their architecture (though I'm sure they've withheld a bit of their special sauce).
In reality, putting aside the millions of $$ needed to pay for the GPUs to train the model, the complexity actually lies in the training data acquisition/cleaning and the infrastructure needed to harness the 1000s of GPUs to train it in a remotely reasonable timeframe.
That being said there are a number of companies (Google, AI21, Cohere, and probably others) who have successfully created large language models like GPT3, so it's definitely not impossible when you have the resources.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#8I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
He's building the model from scratch, as the title suggests. He only trains a small model with 10M parameters on it, something that is feasible with a single GPU. In comparison, GPT-3 has 175B parameters.
> wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training?
It would most certainly be possible for another company to build something very similar (models of similar size have even be released publicly). I'm honestly unsure why Microsoft would rather pay $10B to acquire less than half of OpenAI, as they have the hardware to do it (OpenAI uses MS cloud products.) Must be some business reasons I don't understand. OpenAI definitely has some very talented people working for it, though.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#9I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
He's building the model from scratch, as the title suggests. He only trains a small model with 10M parameters on it, something that is feasible with a single GPU. In comparison, GPT-3 has 175B parameters. > wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training? It would most certainly be possible for another company to build something very similar (model…
Does the time to train the model increase linearly with the number of parameters, or exponentially?
In other words, GPT-3 is 17,500X the number of parameters but does that mean you can train it in 17,500X the amount of time it takes to train the 10M param model?
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#10The char-rnn guy does it again.