Do you have plans to create videos for the llm.c?
Reproducing GPT-2 in llm.c
31–40 of 127 posts
Re: Reproducing GPT-2 in llm.c
#32Earlier quoted context omitted.
Thank you for the effort you put in your educational work, it helped me and others a lot! In fact, i'm training my nanoGPT version right now. :) > Ultimately my interest in llm.c is to have a nice, clean, minimal, super dependency-light repo in direct C/CUDA implementation, which I find aesthetically pleasing. Also, it's awesome that you spend your time on your passion. Any plans on making a video series on llm.c? :D
Yes definitely. Related tweet of mine: https://x.com/karpathy/status/1760388761349927356?lang=en 1. Build the thing 2. Build the ramp Currently on step 1 :). It helps to build it first so you know where you are going, and then you can more easily re-build it when you're vector pointed at the end result.
Everytime you take gardening leave, you build something new and interesting!
Re: Reproducing GPT-2 in llm.c
#33Earlier quoted context omitted.
i suppose you wouldn't be able to use it for external services, but internally, I'm sure you can find some books that fell off the back of a truck...
No reason you can't go external. GPT was trained using ebook torrent sites
OpenAI has enough money to hire lawyers to defend it until the end of time though
Re: Reproducing GPT-2 in llm.c
#34I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.
Considering it takes 8x A100 GPUs (80GB VRAM) to train GPT-2, I think it'll take far more than a single 4090.
Re: Reproducing GPT-2 in llm.c
#35Earlier quoted context omitted.
https://huggingface.co/datasets/HuggingFaceFW/fineweb has 15T cleaned and deduplicated english web data tokens.
Holy crap, Does huggingface charge for bandwidth if you're downloading 45 terabytes??
I believe they are hosting it on Cloudflare who doesn’t charge for egress
Re: Reproducing GPT-2 in llm.c
#36I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.
Considering it takes 8x A100 GPUs (80GB VRAM) to train GPT-2, I think it'll take far more than a single 4090.
The RTX 4090 has about the same BF16 Tensor Core TOPs than the A100, assuming 50% MFU (like the A100 40 GB PCIe) it would take 8x longer on 1 RTX 4090 vs 8x A100 80GB SXM, so 12 hours.
Datasheet here for the TOPs https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvid...
50% MFU should be achievable on the 4090.
Re: Reproducing GPT-2 in llm.c
#37I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.
FWIW, I'm seeing ~318,000 toks/sec throughput on a 4x AMD 7900 XTX machine (less than $4k worth of GPU), using the same settings as in the post (0.5M batch size etc).
Re: Reproducing GPT-2 in llm.c
#38Earlier quoted context omitted.
Holy crap, Does huggingface charge for bandwidth if you're downloading 45 terabytes??
I believe they are hosting it on Cloudflare who doesn’t charge for egress
More specifically, Cloudflare R2 doesn't charge for egress, and Cloudflare doesn't charge for egress to members in the Bandwidth Alliance which include Azure, Google Cloud, Oracle, Alibaba Cloud, and others, though critically not AWS.
They very much do charge egress fees elsewhere.
Re: Reproducing GPT-2 in llm.c
#39Earlier quoted context omitted.
You might have covered this topic before, but I'm curious about the main performance differences between nanoGPT and llm.c. I'm planning to take your "Zero to Hero" course, and I'd like to know how capable the nanoGPT chatbot you'll build is. Is its quality comparable to GPT-2 when used as a chatbot?
Zero To Hero doesn't make it all the way to a chatbot, it stops at pretraining, and even that at a fairly small scale or character-level transformer on TinyShakespeare. I think it's a good conceptual intro but you don't get too too far as a competent chatbot. I think I should be able to improve on this soon.
Thanks! So, you are considering expanding the Zero to Hero series to include building a basic GPT-2 toy chatbot? I believe you mentioned in one of the early lectures that you planned to include building a toy version of Dalle. Do you still have plans for that as well?
Re: Reproducing GPT-2 in llm.c
#40Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!
Ok, we've changed the URL to that from https://twitter.com/karpathy/status/1795484547267834137 above. Thanks!