Karpathy also has a great Recipe for Training Neural Networks:
Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
61–70 of 111 posts
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#62Earlier quoted context omitted.
I gave up and moved to lambdalabs, then they ran out of quota across the board, and now I use a combination of Vast, and Coreweave today.
b-but you work for- why wouldn't they... you know what, never mind.
Oh, I quit recently.
Very surprisingly, I learned it was harder to get access to GPUs at big tech companies outside their dedicated research teams than it was for a scrappy hacker outside on side projects with lots of savings. So I quit to work on those projects. I miss not having a dedicated infra team but don’t miss having to beg for resources. I wanted to use google cloud since I use some of their other services for this, but could not figure out how to get them to increase quotas and take my money. I was willing to pay nearly double the hourly rate I am currently paying at coreweave to use only one cloud provider but they just wouldn’t sell me it
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#63I am a simple man. I see a video post by karpathy, I upvote and watch. I discovered Andrej very recently and I am a huge fan. Kudos to this whole effort! Two ideas -- 1. While these explainers are outstanding -- I can think of supplementary material/presentation that can nicely complement these explanations if they are presented visually. Especially the concepts of multidimensional tensors. Something like what 3B1B (…
In case you're not aware, 3B1B has a Github repo for the engine he uses for the math animations so that others can use it to make similar things: https://github.com/3b1b/manim
There's also a loose group of people already doing the visual learners "explainers" thing over here: https://explorabl.es/ (you can scroll down for links to tools they use to make their explainers).
But yes, I also feel this is an important development and that this should be an ongoing way of teaching people things. Formal education has IMO stalled out around the printing press but there are massive opportunities on computers (and especially on globally networked computers) to take that a step further and leverage the capabilities of computers to make education even more engaging and information-dense.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#64I had the please of working under Andrej a few years ago. It was very interesting the problems we were solving until this other guy that is a billionaire would throw a wrench (and fits) into the whole thing.
[deleted]
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#65I am a simple man. I see a video post by karpathy, I upvote and watch. I discovered Andrej very recently and I am a huge fan. Kudos to this whole effort! Two ideas -- 1. While these explainers are outstanding -- I can think of supplementary material/presentation that can nicely complement these explanations if they are presented visually. Especially the concepts of multidimensional tensors. Something like what 3B1B (…
[deleted]
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#66Earlier quoted context omitted.
He's building the model from scratch, as the title suggests. He only trains a small model with 10M parameters on it, something that is feasible with a single GPU. In comparison, GPT-3 has 175B parameters. > wondering like how hard is it to actually replicate what openAI has done if you had the money to pay for the training? It would most certainly be possible for another company to build something very similar (model…
Does the time to train the model increase linearly with the number of parameters, or exponentially? In other words, GPT-3 is 17,500X the number of parameters but does that mean you can train it in 17,500X the amount of time it takes to train the 10M param model?
On top of what other people have said about parallelism overheads, you normally need more data to train a bigger network and the training time is roughly proportional to network size * training data.
IIRC OpenAI used a million times more data to train GPT3 than karpathy used in this video, so a naive estimate would be that it would take about 20 billion times more compute. This is could be a significant overestimate since Karpathy probably used each bit of the training set more times than openAI did.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#67I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
Yeah, that would be really interesting, open source models with similar quality to GPT that even smaller players could use to train models tailored for their application. Kind of the language equivalent to what the OS community already achieved for image models (eg replicating Dreambooth for StableDiffusion). I think one problem standing in the way of that would be that the computational requirements for language models just seem to be a lot higher.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#68His cs231n course is how I learnt about neutral networks. And now on to learning GPT and Transformers.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#69A bit off topic but, the power of GPT (and DL in general) is in the data. Yet, we’ve allowed private enterprises to control what should be distinctly public goods. I don’t know where we took the wrong turn within the past decade but we desperately need to correct this mistake.
Re: Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy [video]
#70Earlier quoted context omitted.
Same, but I would honestly rather we get away from silicon or anything resembling what we call these "computers" these days. Things are too complicated and we need to truly get back to the basics: rocks on a hill.
Abacus-driven deep learning with trained monkeys flipping gradients.
You don't need to flip gradients, just need to train the monkeys to operate matchboxes: https://en.wikipedia.org/wiki/Matchbox_Educable_Noughts_and_...