61–70 of 127 posts
Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!
Why write in CUDA and not just use PyTorch etc? if performance, how much faster is it, out of curiosity?
“LLM training in simple, pure C/CUDA. There is no need for 245MB of PyTorch or 107MB of cPython. […] A few more words on what I want this repo to be: First, I want llm.c to be a place for education.”
In your opinion is it important for ML engineers to know C?
In your opinion is it important for ML engineers to know C?
I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.
FWIW, I'm seeing ~318,000 toks/sec throughput on a 4x AMD 7900 XTX machine (less than $4k worth of GPU), using the same settings as in the post (0.5M batch size etc).
Earlier quoted context omitted.
Hi Andrej! First, thank you for your teaching, it has helped me a lot, didn't think I'd ever have the chance to say thank you, but here you are and I hope this gets to you! Question - what's a relevant (05-2024) baseline to compare the performance of c code to? Back when you made nanoGPT you were seeing "the file train.py reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of…
The baseline is definitely PyTorch (or JAX), and indeed something like nanoGPT. I just never got nanoGPT "past the finish line" of really crossing the t's and dotting the i's and reproducing the models with as much care as I did now and here in llm.c, and getting to the point where it's a single launch command that just does the thing. I think I'll try to develop the `train_gpt2.py` inside llm.c to be that, so that w…
Earlier quoted context omitted.
Hi Andrej! First, thank you for your teaching, it has helped me a lot, didn't think I'd ever have the chance to say thank you, but here you are and I hope this gets to you! Question - what's a relevant (05-2024) baseline to compare the performance of c code to? Back when you made nanoGPT you were seeing "the file train.py reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of…
The baseline is definitely PyTorch (or JAX), and indeed something like nanoGPT. I just never got nanoGPT "past the finish line" of really crossing the t's and dotting the i's and reproducing the models with as much care as I did now and here in llm.c, and getting to the point where it's a single launch command that just does the thing. I think I'll try to develop the `train_gpt2.py` inside llm.c to be that, so that w…
Earlier quoted context omitted.
Ok, we've changed the URL to that from https://twitter.com/karpathy/status/1795484547267834137 above. Thanks!
sounds good. both work, (though) I think HN has a bit of an anti-twitter bias.
For why I think we have a anti-twitter bias...
Twitter doesn't show replies or any further context without being logged in. Most people will have accounts but I know a lot here deleted theirs or refuse to use it for one reason or another.
Also IMO most here are going to want to read the full source so it just cuts out the middleman. This would usually fall under the "Please submit the original source. If a post reports on something found on another site, submit the latter." guideline which is a little different since the source is yourself, but still the Twitter post doesn't add anything new or novel.
Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!
Ok, we've changed the URL to that from https://twitter.com/karpathy/status/1795484547267834137 above. Thanks!
I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.
I'm not saying this to be rude, but I think you have a deep misunderstanding of how AI training works. You cannot just skip the matrix multiplications necessary to train the model, or get current hardware to do it faster.
https://openaccess.thecvf.com/content_CVPR_2020/papers/Chen_...
https://arxiv.org/abs/2012.03458
https://openaccess.thecvf.com/content/CVPR2021W/MAI/papers/E...