Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reproducing GPT-2 in llm.c

github.com

Post reply on HN
Print view View original

71–80 of 127 posts

Re: Reproducing GPT-2 in llm.c

#71

Post by elicksaur »

vineyardmike wrote: post #14
benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

We won’t ever get there or need to because GPT-4 wasn’t trained on one GPU it was trained on thousands. The (most likely) biggest meaningful difference between -2 and -4 is the number of parameters and the training data/duration. I don’t think you’d really learn much more.

It’s not about learning. It’s about owning. Exactly the reason OpenAI stopped being open. Having GPT-4-quality LLMs created by anyone with a gaming PC would be pretty radical.

Re: Reproducing GPT-2 in llm.c

#73

Post by metadat »

karpathy wrote: post #58
localhost wrote: post #54

Earlier quoted context omitted.

How large is the set of binaries needed to do this training job? The current pytorch + CUDA ecosystem is so incredibly gigantic and manipulating those container images is painful because they are so large. I was hopeful that this would be the beginnings of a much smaller training/fine-tuning stack?

That is 100% my intention and hope and I think we are very close to deleting all of that. Right now on master, I am already only using Python for the tokenization preprocessing. In principle the requirements for llm.c should be extremely minimal. I think this a few days of work that is high on my mind. Biggest problem right now is finding a place that can host the 135GB of tokens for FineWeb100B. Will probably use S3…

Could this be a good case for a torrent?

Re: Reproducing GPT-2 in llm.c

#74

Post by dekhn »

karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

Would you consider switching your interest to protein structure prediction? In particular, the current most advanced model is a closed-source, closed-weights system that was trained on a proprietary hardware. It is intentionally kept that way for now to enable deepmind to commercialize their product.

The goal here isn't to make the best performing model: it's ablation. How much can we remove from protein structure prediction (such as multiple sequence alignments and molecular dynamics, which were two improvements in AF3), while still having a generalized model that can predict novel folds.

Then focus on teaching the minimal necessary math and code to reproduce the results to the larger biological community. All I can say about AF3 is that it literally taught me that everything I learned about protein structure prediction in the last 30 years was misguided, or outright wrong.

Don't worry about drug discovery or any of the hard stuff. Just continue to show that all that's required to predict novel structures is the existing PDB.

Re: Reproducing GPT-2 in llm.c

#75

Post by anthonix1 »

pama wrote: post #64
anthonix1 wrote: post #37

Earlier quoted context omitted.

FWIW, I'm seeing ~318,000 toks/sec throughput on a 4x AMD 7900 XTX machine (less than $4k worth of GPU), using the same settings as in the post (0.5M batch size etc).

Did you reproduce the evaluation as well?

It converges similarly on smaller datasets.

About to kick off a training from scratch run on the same fineweb-10B, which at 324k toks/sec should take about 8.6 hours. And with my kWh cost, that is about $2.50 cost to train.

Will report back tomorrow when the training has finished..

Re: Reproducing GPT-2 in llm.c

#76

Post by karpathy »

pests wrote: post #67
karpathy wrote: post #45

Earlier quoted context omitted.

sounds good. both work, (though) I think HN has a bit of an anti-twitter bias.

First, love the videos and other work you've been doing. The micrograd videos are a great way to show people this is all math in the end, and I've linked to specific timestamps in that video and others more times than I can count. For why I think we have a anti-twitter bias... Twitter doesn't show replies or any further context without being logged in. Most people will have accounts but I know a lot here deleted thei…

fwiw I totally understand the sentiment! it's actually a bit sad to me that so much of our content is moving from the shared, open web to platforms like twitter, unfortunately there seems to be too much value add around built-in discoverability, comments, ease of authoring, for many people revenue sharing, etc.

Re: Reproducing GPT-2 in llm.c

#77

Post by vineyardmike »

elicksaur wrote: post #71
vineyardmike wrote: post #14

Earlier quoted context omitted.

We won’t ever get there or need to because GPT-4 wasn’t trained on one GPU it was trained on thousands. The (most likely) biggest meaningful difference between -2 and -4 is the number of parameters and the training data/duration. I don’t think you’d really learn much more.

It’s not about learning. It’s about owning. Exactly the reason OpenAI stopped being open. Having GPT-4-quality LLMs created by anyone with a gaming PC would be pretty radical.

And you won’t get there. Those models are far too large for a 2024 GPU. Llama-3 70b is arguably close to GPT-4 but is still too large for gaming GPUs (and probably for many years of GPU updates)

Re: Reproducing GPT-2 in llm.c

#79

Post by LorenzoGood »

karpathy wrote: post #24
lagrange77 wrote: post #21

Earlier quoted context omitted.

Thank you for the effort you put in your educational work, it helped me and others a lot! In fact, i'm training my nanoGPT version right now. :) > Ultimately my interest in llm.c is to have a nice, clean, minimal, super dependency-light repo in direct C/CUDA implementation, which I find aesthetically pleasing. Also, it's awesome that you spend your time on your passion. Any plans on making a video series on llm.c? :D

Yes definitely. Related tweet of mine: https://x.com/karpathy/status/1760388761349927356?lang=en 1. Build the thing 2. Build the ramp Currently on step 1 :). It helps to build it first so you know where you are going, and then you can more easily re-build it when you're vector pointed at the end result.

I love when you leave your job.

Re: Reproducing GPT-2 in llm.c

#80

Post by ls612 »

Is this the sort of thing that a person with curiosity and a 4090 could do? It says he used 8xA100s in the cloud to do this but is it just a matter of the 4090 going 8x slower or will memory constraints kill the whole endeavour?
Post reply on HN