Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reproducing GPT-2 in llm.c

github.com

Post reply on HN
Print view View original

21–30 of 127 posts

Re: Reproducing GPT-2 in llm.c

#21

Post by lagrange77 »

karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

Thank you for the effort you put in your educational work, it helped me and others a lot! In fact, i'm training my nanoGPT version right now. :)

> Ultimately my interest in llm.c is to have a nice, clean, minimal, super dependency-light repo in direct C/CUDA implementation, which I find aesthetically pleasing.

Also, it's awesome that you spend your time on your passion.

Any plans on making a video series on llm.c? :D

Re: Reproducing GPT-2 in llm.c

#22

Post by meiraleal »

CamperBob2 wrote: post #19
meiraleal wrote: post #13

Earlier quoted context omitted.

I'm okay with paying for datasets

Depends on how the courts rule. If the copyright maximalists prevail, only the wealthiest entities will be able to afford to license a useful data set. Paradoxically enough, this is the outcome that most "Hacker News" denizens seem to be rooting for.

I'd still get most of my dataset from torrent but I could pay for specific things like high quality source code.

Re: Reproducing GPT-2 in llm.c

#23

Post by GaggiX »

ravetcofx wrote: post #7
benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

Accessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4

https://huggingface.co/datasets/HuggingFaceFW/fineweb has 15T cleaned and deduplicated english web data tokens.

Re: Reproducing GPT-2 in llm.c

#24

Post by karpathy »

lagrange77 wrote: post #21
karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

Thank you for the effort you put in your educational work, it helped me and others a lot! In fact, i'm training my nanoGPT version right now. :) > Ultimately my interest in llm.c is to have a nice, clean, minimal, super dependency-light repo in direct C/CUDA implementation, which I find aesthetically pleasing. Also, it's awesome that you spend your time on your passion. Any plans on making a video series on llm.c? :D

Yes definitely. Related tweet of mine:

https://x.com/karpathy/status/1760388761349927356?lang=en

1. Build the thing

2. Build the ramp

Currently on step 1 :). It helps to build it first so you know where you are going, and then you can more easily re-build it when you're vector pointed at the end result.

Re: Reproducing GPT-2 in llm.c

#25

Post by 363849473754 »

karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

You might have covered this topic before, but I'm curious about the main performance differences between nanoGPT and llm.c. I'm planning to take your "Zero to Hero" course, and I'd like to know how capable the nanoGPT chatbot you'll build is. Is its quality comparable to GPT-2 when used as a chatbot?

Re: Reproducing GPT-2 in llm.c

#26

Post by bilsbie »

sturza wrote: post #10
karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

Do you think grokking leads to proper generalized reasoning? https://arxiv.org/abs/2405.15071

Any tips on understanding grokking? I’m not following that paper.

Re: Reproducing GPT-2 in llm.c

#27

Post by karpathy »

363849473754 wrote: post #25
karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

You might have covered this topic before, but I'm curious about the main performance differences between nanoGPT and llm.c. I'm planning to take your "Zero to Hero" course, and I'd like to know how capable the nanoGPT chatbot you'll build is. Is its quality comparable to GPT-2 when used as a chatbot?

Zero To Hero doesn't make it all the way to a chatbot, it stops at pretraining, and even that at a fairly small scale or character-level transformer on TinyShakespeare. I think it's a good conceptual intro but you don't get too too far as a competent chatbot. I think I should be able to improve on this soon.

Re: Reproducing GPT-2 in llm.c

#28

Post by lagrange77 »

karpathy wrote: post #24
lagrange77 wrote: post #21

Earlier quoted context omitted.

Thank you for the effort you put in your educational work, it helped me and others a lot! In fact, i'm training my nanoGPT version right now. :) > Ultimately my interest in llm.c is to have a nice, clean, minimal, super dependency-light repo in direct C/CUDA implementation, which I find aesthetically pleasing. Also, it's awesome that you spend your time on your passion. Any plans on making a video series on llm.c? :D

Yes definitely. Related tweet of mine: https://x.com/karpathy/status/1760388761349927356?lang=en 1. Build the thing 2. Build the ramp Currently on step 1 :). It helps to build it first so you know where you are going, and then you can more easily re-build it when you're vector pointed at the end result.

That's fantastic. My gradient field is pointing towards it.

Thank you again!

Re: Reproducing GPT-2 in llm.c

#29

Post by sturza »

bilsbie wrote: post #26
sturza wrote: post #10

Earlier quoted context omitted.

Do you think grokking leads to proper generalized reasoning? https://arxiv.org/abs/2405.15071

Any tips on understanding grokking? I’m not following that paper.

Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. Overfitting and being cool about it and some new behavior might emerge.

Re: Reproducing GPT-2 in llm.c

#30

Post by ravetcofx »

GaggiX wrote: post #23
ravetcofx wrote: post #7

Earlier quoted context omitted.

Accessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4

https://huggingface.co/datasets/HuggingFaceFW/fineweb has 15T cleaned and deduplicated english web data tokens.

Holy crap, Does huggingface charge for bandwidth if you're downloading 45 terabytes??
Post reply on HN