Thank you so much for this! It is so impressive and I'm sure it took a lot of hard work!
Is it able to re-write articles? And where could I find a guide on how to train it?
NanoGPT
221–230 of 334 posts
Re: NanoGPT
#222Earlier quoted context omitted.
> very little money in optimizing deep learning Oh - there are a lot of people working on optimizing AI. Amongst hobbyists, academia, and corporations alike. The thing is, if you come up with a neat optimization that saves 30% of compute for the same results, typically instead of reducing your compute budget 30%, you instead increase your model/data size 30% and get better results.
Jevon's paradox of data and AI. The more efficiently data is used, the more demand their is for data.
Any state of the art model takes about three weeks to train.
Re: NanoGPT
#223Earlier quoted context omitted.
The repo seems to imply that it matches GPT-2, so I imagine any analyses of GPT-2 will give you a good idea.
I’m not easily finding GPT-2 use cases. Any query guidance?
The GPT family of models shines above 100B parameters. Almost nobody uses GPT2 today. It's too weak.
If you want to go with <1B model, you use a BERT which is bidirectional or a T5 that is easier to fine-tune on other tasks.
Re: NanoGPT
#224I could not find any sample (prompt and results). Can anyone provide samples of it's quality, even if it is in a narrow field of knowledge or specific use case? I tried GPT2, GPT-J 6B and GPT-NeoX 20B (implementation by Fabrice Bellard at textsynth.com/playground.html) but I could not find any production-quality scenario yet, only cherry-picked simple cases.
At this model size quality is not worth discussing. It is clearly another league from GPT-3.
Re: NanoGPT
#225Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.
Re: NanoGPT
#226Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.
I've found all of your code and lessons on youtube so incredibly useful. You're a wonderful teacher and I really appreciate all the work you've done with this!
Re: NanoGPT
#227This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.
Well, he does include instructions for running it on a personal computer, which looks like what I'm gonna be doing next week.
Besides the rental options discussed below these nvidia boxen don't look too big so either used ones will be available for cheap relatively soon, or you could just locate and liberate one in Promethean fashion.
Re: NanoGPT
#228Earlier quoted context omitted.
I’m all for thanking open source contributors, but your excessively prostrating wording is a bit much for me.
[flagged]
can't tell if you're making some kind of clever quip or if this is some random spambot just entering a random reverse DNS lookup line.
Re: NanoGPT
#229Doesn't huggingface have dozens of freely available pretrained models like this (including various sized implementations of GPT2) and isn't the source available on most if you wanted to train them yourself? All I see in the comments is praise for the author as a person, so just wondering what's unique about this that's not available elsewhere? 730 upvotes and counting, assuming I'm missing something...
[flagged]
Re: NanoGPT
#230Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.
Your tutorials are effective and concise. Thank you for them! Accessible, from-scratch knowledge on these topics is essential at this time in history and you're really making a dent in that problem.