Karpathy's videos (and blogs) are excellent. I wonder how history will reflect on his time at Tesla, however.
As if history will care?
Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
21–30 of 111 posts
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#22I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
He's building the model from scratch, as the title suggests. He only trains a small model with 10M parameters on it, something that is feasible with a single GPU. In comparison, GPT-3 has 175B parameters. > wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training? It would most certainly be possible for another company to build something very similar (model…
>I'm honestly unsure why Microsoft would rather pay $10B to acquire less than half of OpenAI, as they have the hardware to do it (OpenAI uses MS cloud products.)
Because the hardware is the least interesting part of it?
Microsoft buys the know-how, the talent, and perhaps some patents, but most importantly the GPT brand name...
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#23This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.
Andrej's "Building Makemore" series is exactly this, it includes a wonderful lecture where he computes all the gradients for a simple network by hand and compares them against the values produced by torch's autograd.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#24Earlier quoted context omitted.
He's building the model from scratch, as the title suggests. He only trains a small model with 10M parameters on it, something that is feasible with a single GPU. In comparison, GPT-3 has 175B parameters. > wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training? It would most certainly be possible for another company to build something very similar (model…
Does the time to train the model increase linearly with the number of parameters, or exponentially? In other words, GPT-3 is 17,500X the number of parameters but does that mean you can train it in 17,500X the amount of time it takes to train the 10M param model?
In theory it should be linear, however, the parallelization is not perfect and some overlapping parts of gradients are computed on multiple GPUs at the same time so expect some constant factor slowdown on average.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#25This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.
That would be next to impossible to do even in a semester long course...
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#26Earlier quoted context omitted.
Same, but one that doesn't assume python or an operating system.
Same, but I would honestly rather we get away from silicon or anything resembling what we call these "computers" these days. Things are too complicated and we need to truly get back to the basics: rocks on a hill.
Abacus-driven deep learning with trained monkeys flipping gradients.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#27Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#28Earlier quoted context omitted.
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…
On number 2, even if you are John Carmack you may have trouble getting the right people on the phone. https://twitter.com/id_aa_carmack/status/1305967411749892098... Anyone at Google Cloud out there? It seems I can't get my GPU quota raised to 40 x V100 as an independent researcher. I was told that setting up a website would help, but I would rather not. I can pay the bills...
If you're actually an independent researcher, sometimes you can find professors at universities or national labs that are willing to help out in exchange for credits on the paper. I've had success at [redacted] labs in the New Mexico region as well as folks from my previous university. The trick is asking people who do research that's sort of adjacent to your field.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#29I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
He's building the model from scratch, as the title suggests. He only trains a small model with 10M parameters on it, something that is feasible with a single GPU. In comparison, GPT-3 has 175B parameters. > wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training? It would most certainly be possible for another company to build something very similar (model…
They are buying the talent like they would when they buy any company. They are certainly not buying a single trained model.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#30Karpathy's videos (and blogs) are excellent. I wonder how history will reflect on his time at Tesla, however.
What do you mean by "however"? He built the entire FSD architecture - in less time and with fewer people than Google, Apple, and god knows who else. He's likely saved more lives than most doctors at this point...