Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reproducing GPT-2 in llm.c

github.com

Post reply on HN
Print view View original

91–100 of 127 posts

Re: Reproducing GPT-2 in llm.c

#91

Post by adeptima »

Andrej Karpathy karpathy is a magician!

But being the coolest kid on the block with pure C/CUDA implementation is not enough https://github.com/karpathy/llm.c

Studying a baby Llama 2 model source code in pure Mojo is the next level https://github.com/tairov/llama2.mojo https://github.com/tairov/llama2.mojo/blob/master/llama2.moj...

Mojo Lang - Tomorrow's High Performance Python? (with Chris Lattner) https://www.youtube.com/watch?v=JRcXUuQYR90

Andrej Karpathy and Chris Lattner collab is on my wishlist ;)

Re: Reproducing GPT-2 in llm.c

#92

Post by adeptima »

natsucks wrote: post #57

In your opinion is it important for ML engineers to know C?

Spend one year to study multiple languages - bash, C, C++, Go, Python ... and even Mojo or Rust. 10-20 hours a week. Being able to read top programming languages is the best investment I ever made. You will become fearless and can see the matrix ;)

Re: Reproducing GPT-2 in llm.c

#93

Post by sytelus »

karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

So, NanoGPT took 1.8 days on 8xA100 for 124M model training on 30.7B tokens using flash attention. This would translate to 14.4hr for 10B tokens. With llm.c it is ~1.5 hr which is almost 10X speedup!

Does this look ballpark correct? Is there any summary of where majority of this improvement comes from?

Re: Reproducing GPT-2 in llm.c

#94

Post by wrboyce »

dang wrote: post #81
wrboyce wrote: post #69

Earlier quoted context omitted.

Could you mention what the link has been changed from too? Sometimes it helps with context when reading the comments. Thanks!

I agree that it helps! but I did mention it, no? Admittedly "to that from" is a bit of an awkward construction

facepalm I’d had a few whiskies and misread your comment. Sorry about that!

Re: Reproducing GPT-2 in llm.c

#95

Post by sebzim4500 »

CamperBob2 wrote: post #11
ravetcofx wrote: post #7

Earlier quoted context omitted.

Accessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4

Someone will come along and say "Why don't you just mirror Anna's Archive?" in 3...2...1...

I think between Anna's Archive, fineweb and as many github repos as you can scrape you can get a pretty decent dataset.

I doubt Anna's Archive would produce a good model on its own though.

Re: Reproducing GPT-2 in llm.c

#96

Post by benterix »

Invictus0 wrote: post #18
benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

I'm not saying this to be rude, but I think you have a deep misunderstanding of how AI training works. You cannot just skip the matrix multiplications necessary to train the model, or get current hardware to do it faster.

No offence taken! As far as my (shallow!) understanding goes, the main challenge is the need for many GPUs with huge amounts of memory, and it still takes ages to train the model. So regarding the use of consumer GPUs, some work has been done already, and I've seen some setups where people combine of these and are successful. As for the the other aspects, maybe at some point we distill what is really needed to a smaller but excellent dataset that would give similar results in the final models.

Re: Reproducing GPT-2 in llm.c

#97

Post by verticalscaler »

andersa wrote: post #86
ravetcofx wrote: post #30

Earlier quoted context omitted.

Holy crap, Does huggingface charge for bandwidth if you're downloading 45 terabytes??

Fun trivia: downloading 45TB costs about $60, according to Cloudflare.

That's what Cloudflare charges. It costs them around 6 cents.

Re: Reproducing GPT-2 in llm.c

#98

Post by treme »

dekhn wrote: post #74
karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

Would you consider switching your interest to protein structure prediction? In particular, the current most advanced model is a closed-source, closed-weights system that was trained on a proprietary hardware. It is intentionally kept that way for now to enable deepmind to commercialize their product. The goal here isn't to make the best performing model: it's ablation. How much can we remove from protein structure pr…

lol I appreciate your effort to guide his genius towards 'max human good'
Post reply on HN