NanoGPT: The simplest, fastest repository for training medium-sized GPTs
1–10 of 23 posts
Re: NanoGPT: The simplest, fastest repository for training medium-sized GPTs
#2Previous discussion: https://news.ycombinator.com/item?id=34336386 (1532 points | Jan 11, 2023 | 320 comments)
Re: NanoGPT: The simplest, fastest repository for training medium-sized GPTs
#3See also (from the same author) https://github.com/karpathy/llm.c — "LLMs in simple, pure C/CUDA with no need for 245MB of PyTorch or 107MB of cPython."
Re: NanoGPT: The simplest, fastest repository for training medium-sized GPTs
#4This guy is awesome
Re: NanoGPT: The simplest, fastest repository for training medium-sized GPTs
#5It takes a bit of effort to setup and train a proper GPT-2 model, especially if you are not familiar with GPU drivers and python environment.
Also don't try to train GPT-2 on your own machine as it takes days, even with a good gaming GPU.
If you are interested in trying it out, but don't have the right GPU or OS, you can check out the guide I wrote on how I did it on Azure with T4 GPU instance:
https://16x.engineer/2023/12/29/nanoGPT-azure-T4-ubuntu-guid...
Re: NanoGPT: The simplest, fastest repository for training medium-sized GPTs
#6The fact that Andrej Karpathy is part of OpenAI is giving me at least a bit of hope that the company is not going to ruin society as we know it…
Re: NanoGPT: The simplest, fastest repository for training medium-sized GPTs
#7Would any GPU - even an old one - work for training these models?
I have a bunch of little home apps/ideas for some specific training I'd like to do, but don't currently have any recent GPU of note. Just a few old ones in my attic somewhere.
EDIT: ah someone has posted a blog post in the comments with some info
EDIT2: to be clearer I was talking about finetuning. I assume thats quicker/less intensive. I really need to checkout some online courses I think.
Re: NanoGPT: The simplest, fastest repository for training medium-sized GPTs
#8The fact that Andrej Karpathy is part of OpenAI is giving me at least a bit of hope that the company is not going to ruin society as we know it…
he…isn’t?
Re: NanoGPT: The simplest, fastest repository for training medium-sized GPTs
#9The fact that Andrej Karpathy is part of OpenAI is giving me at least a bit of hope that the company is not going to ruin society as we know it…
He left in Feb 2024.
https://x.com/karpathy/status/1757600075281547344?t=gpYK73N5...