Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reproducing GPT-2 in llm.c

github.com

Post reply on HN
Print view View original

101–110 of 127 posts

Re: Reproducing GPT-2 in llm.c

#101

Post by akkishore »

Hi Andrej,

Huge fan of all the work you do. Wanted to understand something fundamental and whom better to ask than you: Whats so special about the transformer architecture that its able to predict the next token so beautifully understanding all the intricate previous token relationships? I understand Attention but what so special about this architecture that no other architectures are able to "attend" appropriately to previous tokens? Being a CS guy, its really hard for me to fathom that we have not yet created another architecture which can perform similarly.

Re: Reproducing GPT-2 in llm.c

#102

Post by huac »

sabareesh wrote: post #60
benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

Well here is a comment on 4090 https://github.com/karpathy/llm.c/discussions/481#discussion...

25% MFU :( maybe because of the P2P nerf?

Re: Reproducing GPT-2 in llm.c

#103

Post by anthonix1 »

pama wrote: post #64
anthonix1 wrote: post #37

Earlier quoted context omitted.

FWIW, I'm seeing ~318,000 toks/sec throughput on a 4x AMD 7900 XTX machine (less than $4k worth of GPU), using the same settings as in the post (0.5M batch size etc).

Did you reproduce the evaluation as well?

So... successfully reproduced in ~8.75 hours, taking about 18 kWh / $2.70

The first run actually failed at step 3000 or so, and I realized I had a bug in my attention / matmul kernels, but after fixing that and restarting it worked great

[1] https://github.com/anthonix/llm.c

Re: Reproducing GPT-2 in llm.c

#104

Post by anthonix1 »

auspiv wrote: post #34
benterix wrote: post #3

I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.

Considering it takes 8x A100 GPUs (80GB VRAM) to train GPT-2, I think it'll take far more than a single 4090.

Nah, I reproduced on 4x 7900 XTX machine in 8.75 hours, so a single 7900 XTX (costs less than $1k) could do it in under 24 hours. Was hitting 55.4% MFU.

Re: Reproducing GPT-2 in llm.c

#105

Post by anthonix1 »

huac wrote: post #102
sabareesh wrote: post #60

Earlier quoted context omitted.

Well here is a comment on 4090 https://github.com/karpathy/llm.c/discussions/481#discussion...

25% MFU :( maybe because of the P2P nerf?

Maybe get a 7900 XTX. 122 TFLOPS of BF16/FP16 for less than $1k and I'm getting 55.4% MFU

Re: Reproducing GPT-2 in llm.c

#107

Post by vineyardmike »

elicksaur wrote: post #85
vineyardmike wrote: post #77

Earlier quoted context omitted.

And you won’t get there. Those models are far too large for a 2024 GPU. Llama-3 70b is arguably close to GPT-4 but is still too large for gaming GPUs (and probably for many years of GPU updates)

“You won’t get there” is a pretty vast statement for all of the future. Two fairly reasonable predictions: 1) the compute needed to get GPT4 performance will decrease. 2) the compute on consumer GPUs will increase. At some point they cross, and you will be able to run a GPT4-quality LLM on a consumer GPU. At some point after that, you’ll be able to run a GPT4-quality LLM on a 2024 consumer GPU if you can find one. Im…

> 1) the compute needed to get GPT4 performance will decrease. 2) the compute on consumer GPUs will increase.

I’m assuming we’re just talking inference here…

Sure compute abilities for consumers will increase but the original comment had a fixed GPU - the 4090. I can already eke out LLama3:8b on my MacBook Air, and Apple will sell you a laptop capable of running the full sized LLama.

There is a direct correlation between parameters and “knowledge” for an LM. There’s some open questions as to density (LLaMa3 specifically challenged previous assumptions) but it seems implausible to fit an equivalent model as GPT4 into 24gb vram. Just like compression, you can’t shrink forever.

GPT-4 and GPT-2 are pretty similar architecturally (I assume). So if abilities don’t matter, we can already run GPT-2 so we’re basically there for 4.

Re: Reproducing GPT-2 in llm.c

#108

Post by nwsm »

groby_b wrote: post #51
CamperBob2 wrote: post #19

Earlier quoted context omitted.

Depends on how the courts rule. If the copyright maximalists prevail, only the wealthiest entities will be able to afford to license a useful data set. Paradoxically enough, this is the outcome that most "Hacker News" denizens seem to be rooting for.

It's almost as if people believe in fairness and compensating people for their work. Also, it's worth noting that this is only true as long as we're stuck in the "must train on the entire sum total of human output ever created" local minimum for machine learning. Given that most biological entities learn with much less data, this might well be the thing that prods ML research to using an approach that isn't "IDK, buy…

> It's almost as if people believe in fairness and compensating people for their work.

Yet in this case we are talking about compensating the compilers/massagers/owners of the datasets, not the original authors from wherever the data was originally scraped.

Re: Reproducing GPT-2 in llm.c

#109

Post by anthonix1 »

0x1ceb00da wrote: post #90
karpathy wrote: post #4

Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!

Hi. Is it possible to somehow run llm.c on an amd gpu?

Yeah, I just reproduced the GPT2 from scratch results in 8.75 hours on 4x 7900 XTX. The fork is here: https://github.com/anthonix/llm.c

Re: Reproducing GPT-2 in llm.c

#110

Post by sabareesh »

huac wrote: post #102
sabareesh wrote: post #60

Earlier quoted context omitted.

Well here is a comment on 4090 https://github.com/karpathy/llm.c/discussions/481#discussion...

25% MFU :( maybe because of the P2P nerf?

This much bigger model (500M), P2P is enabled via Mailbox. It is expected because of memory to compute ratio
Post reply on HN