31–40 of 111 posts
Earlier quoted context omitted.
Same, but I would honestly rather we get away from silicon or anything resembling what we call these "computers" these days. Things are too complicated and we need to truly get back to the basics: rocks on a hill.
Abacus-driven deep learning with trained monkeys flipping gradients.
We'll selectively breed humans until one can just compute it in...meat[0].
This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.
This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.
Andrej's "Building Makemore" series is exactly this, it includes a wonderful lecture where he computes all the gradients for a simple network by hand and compares them against the values produced by torch's autograd.
This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.
The entire series is at https://karpathy.ai/zero-to-hero.html, first video is literally from scratch (i.e. just Python, the only external dependency is for plotting graphs if I recall correctly).
This is really great, thank you. I would love to see a real "from scratch" that doesn't use torch.py et. al., though.
That would be next to impossible to do even in a semester long course...
Karpathy's videos (and blogs) are excellent. I wonder how history will reflect on his time at Tesla, however.
What do you mean by "however"? He built the entire FSD architecture - in less time and with fewer people than Google, Apple, and god knows who else. He's likely saved more lives than most doctors at this point...
Karpathy's videos (and blogs) are excellent. I wonder how history will reflect on his time at Tesla, however.
What do you mean by "however"? He built the entire FSD architecture - in less time and with fewer people than Google, Apple, and god knows who else. He's likely saved more lives than most doctors at this point...
which I'm sure was more a marketing error than anything else.
Earlier quoted context omitted.
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…
On number 2, even if you are John Carmack you may have trouble getting the right people on the phone. https://twitter.com/id_aa_carmack/status/1305967411749892098... Anyone at Google Cloud out there? It seems I can't get my GPU quota raised to 40 x V100 as an independent researcher. I was told that setting up a website would help, but I would rather not. I can pay the bills...
I might be too new to this area -- but is this actually explaining how to create like a small version of the actual trained model -- not like "using the trained model for X"? like I can imagine in the future people won't start from pure scratch, there will be building blocks that everybody starts from, but mostly just wondering like how hard is it to actually replicate what openAI has done if you had the money to pay…
rough steps: 1. collect a very large dataset, see: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla... . scrape, de-duplicate, clean, wrangle. this is a lot of work regardless of $. 2. get on a call with the sales teams of major cloud providers to procure a few thousands GPUs and enter into too long contracts. 3. "pretrain" a GPT. one common way to do this atm is to create your own exotic fork of Megatron…
The pretrained model is already available.