Reproducing GPT-2 in llm.c • Hacker News Board

Re: Reproducing GPT-2 in llm.c

#11

Post by CamperBob2 » Tue, May 28, 2024, 5:15 PM UTC

ravetcofx wrote: post #7

benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

Accessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4

Someone will come along and say "Why don't you just mirror Anna's Archive?" in 3...2...1...

Re: Reproducing GPT-2 in llm.c

#12

Post by exe34 » Tue, May 28, 2024, 5:15 PM UTC

ravetcofx wrote: post #7

benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

Accessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4

i suppose you wouldn't be able to use it for external services, but internally, I'm sure you can find some books that fell off the back of a truck...

Re: Reproducing GPT-2 in llm.c

#13

Post by meiraleal » Tue, May 28, 2024, 5:16 PM UTC

ravetcofx wrote: post #7

benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

Accessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4

I'm okay with paying for datasets

Re: Reproducing GPT-2 in llm.c

#14

Post by vineyardmike » Tue, May 28, 2024, 5:17 PM UTC

benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

We won’t ever get there or need to because GPT-4 wasn’t trained on one GPU it was trained on thousands. The (most likely) biggest meaningful difference between -2 and -4 is the number of parameters and the training data/duration. I don’t think you’d really learn much more.

Re: Reproducing GPT-2 in llm.c

#15

Post by karpathy » Tue, May 28, 2024, 5:18 PM UTC

ngiyabonga wrote: post #9

karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

Hi Andrej! First, thank you for your teaching, it has helped me a lot, didn't think I'd ever have the chance to say thank you, but here you are and I hope this gets to you! Question - what's a relevant (05-2024) baseline to compare the performance of c code to? Back when you made nanoGPT you were seeing "the file train.py reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of…

The baseline is definitely PyTorch (or JAX), and indeed something like nanoGPT. I just never got nanoGPT "past the finish line" of really crossing the t's and dotting the i's and reproducing the models with as much care as I did now and here in llm.c, and getting to the point where it's a single launch command that just does the thing.

I think I'll try to develop the `train_gpt2.py` inside llm.c to be that, so that we have the two implementations exactly side by side, and it's all nice and comparable.

The C/CUDA code is currently a little bit faster than PyTorch (last time I measured ~2 weeks ago it was about 6% faster), and I think we can push this further. This is done by manually hard-coding a bunch of fusions/optimizations that are non-trivial for torch.compile to find (e.g. our FusedClassifier). But PyTorch has some pending work/PRs that will also speed up their side a lot.

Ultimately my interest in llm.c is to have a nice, clean, minimal, super dependency-light repo in direct C/CUDA implementation, which I find aesthetically pleasing. And on top of that, educational, i.e. using all of the above as an endpoint of an intro LLM course.

Re: Reproducing GPT-2 in llm.c

#16

Post by espadrine » Tue, May 28, 2024, 5:19 PM UTC

karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

How big of a perf improvement would result from using the architectural tweaks that Llama3 and others have put in place since GPT-2?

Re: Reproducing GPT-2 in llm.c

#17

Post by HeatrayEnjoyer » Tue, May 28, 2024, 5:20 PM UTC

exe34 wrote: post #12

ravetcofx wrote: post #7

Earlier quoted context omitted.

Accessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4

i suppose you wouldn't be able to use it for external services, but internally, I'm sure you can find some books that fell off the back of a truck...

No reason you can't go external. GPT was trained using ebook torrent sites

Re: Reproducing GPT-2 in llm.c

#18

Post by Invictus0 » Tue, May 28, 2024, 5:21 PM UTC

benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

I'm not saying this to be rude, but I think you have a deep misunderstanding of how AI training works. You cannot just skip the matrix multiplications necessary to train the model, or get current hardware to do it faster.

Re: Reproducing GPT-2 in llm.c

#19

Post by CamperBob2 » Tue, May 28, 2024, 5:27 PM UTC

meiraleal wrote: post #13

ravetcofx wrote: post #7

Earlier quoted context omitted.

Accessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4

I'm okay with paying for datasets

Depends on how the courts rule. If the copyright maximalists prevail, only the wealthiest entities will be able to afford to license a useful data set.

Paradoxically enough, this is the outcome that most "Hacker News" denizens seem to be rooting for.

Re: Reproducing GPT-2 in llm.c

#20

Post by karpathy » Tue, May 28, 2024, 5:41 PM UTC

espadrine wrote: post #16

karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

How big of a perf improvement would result from using the architectural tweaks that Llama3 and others have put in place since GPT-2?

My understanding and suspicion is mostly less than you think. Llama 3 architecture has the following changes on GPT-2:

1. delete the absolute positional encoding and replace with RoPE

2. delete all biases in all layers (in LayerNorms, they turn into RMSNorm)

3. GeLU -> SwiGLU non-linearity in the MLP

4. longer context length

5. architecture hyperparameter changes, e.g. slightly different aspect ratios

And there was a paper that I can't find the reference to anymore that claimed that if you train long enough, the gap becomes even lower. Possibly because the absolutely positional encoding has enough time to train more fully, where as the RoPE layer benefits from the "inductive bias" it adds in the earlier stages of training.

But I don't have full confidence on the above claim, maybe someone has tried or has better/concrete reference.