Reproducing GPT-2 in llm.c

#1

Post by tosh » Tue, May 28, 2024, 3:58 PM UTC

Re: Reproducing GPT-2 in llm.c

#2

Post by indigodaddy » Tue, May 28, 2024, 4:40 PM UTC

Looks like this is re: training, but wonder how inference would be on some garbage older machine with no GPU on this model?

Re: Reproducing GPT-2 in llm.c

#3

Post by benterix » Tue, May 28, 2024, 4:40 PM UTC

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090."

Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

Re: Reproducing GPT-2 in llm.c

#4

Post by karpathy » Tue, May 28, 2024, 4:44 PM UTC

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481

Happy to answer questions!

Re: Reproducing GPT-2 in llm.c

#5

Post by [deleted] » Tue, May 28, 2024, 4:49 PM UTC

[deleted]

Re: Reproducing GPT-2 in llm.c

#6

Post by [deleted] » Tue, May 28, 2024, 4:50 PM UTC

[deleted]

Re: Reproducing GPT-2 in llm.c

#7

Post by ravetcofx » Tue, May 28, 2024, 4:55 PM UTC

benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

Accessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4

Re: Reproducing GPT-2 in llm.c

#8

Post by 1024core » Tue, May 28, 2024, 4:57 PM UTC

karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

Thank you, from an appreciative reader!

Re: Reproducing GPT-2 in llm.c

#9

Post by ngiyabonga » Tue, May 28, 2024, 5:01 PM UTC

karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

Hi Andrej!

First, thank you for your teaching, it has helped me a lot, didn't think I'd ever have the chance to say thank you, but here you are and I hope this gets to you!

Question - what's a relevant (05-2024) baseline to compare the performance of c code to? Back when you made nanoGPT you were seeing "the file train.py reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in about 4 days of training". So twice the memory on the c node, but unsure of data size /epochs, any other details I may be missing. I.e. what's the net uplift of running c vs "legacy" torch code?

Thanks again for everything.

Re: Reproducing GPT-2 in llm.c

#10

Post by sturza » Tue, May 28, 2024, 5:07 PM UTC

karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

Do you think grokking leads to proper generalized reasoning? https://arxiv.org/abs/2405.15071