Can it be done in rust ?
Reproducing GPT-2 in llm.c
41–50 of 127 posts
Re: Reproducing GPT-2 in llm.c
#42Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!
Why write in CUDA and not just use PyTorch etc?
if performance, how much faster is it, out of curiosity?
Re: Reproducing GPT-2 in llm.c
#43Time for llm videos!
Re: Reproducing GPT-2 in llm.c
#44Earlier quoted context omitted.
You might have covered this topic before, but I'm curious about the main performance differences between nanoGPT and llm.c. I'm planning to take your "Zero to Hero" course, and I'd like to know how capable the nanoGPT chatbot you'll build is. Is its quality comparable to GPT-2 when used as a chatbot?
Zero To Hero doesn't make it all the way to a chatbot, it stops at pretraining, and even that at a fairly small scale or character-level transformer on TinyShakespeare. I think it's a good conceptual intro but you don't get too too far as a competent chatbot. I think I should be able to improve on this soon.
Please do! It's a fantastic series!
Re: Reproducing GPT-2 in llm.c
#45Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!
Ok, we've changed the URL to that from https://twitter.com/karpathy/status/1795484547267834137 above. Thanks!
sounds good. both work, (though) I think HN has a bit of an anti-twitter bias.
Re: Reproducing GPT-2 in llm.c
#46I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.
I'm not saying this to be rude, but I think you have a deep misunderstanding of how AI training works. You cannot just skip the matrix multiplications necessary to train the model, or get current hardware to do it faster.
was the first sentence really necessary? The second sentence seems fine by itself.
Re: Reproducing GPT-2 in llm.c
#47Looks like this is re: training, but wonder how inference would be on some garbage older machine with no GPU on this model?
Last time I tried GPT-2 on CPU (which I think was shortly before chatGPT was launched), I was getting about 0.2 tokens/sec. CPU utilization was low though, so running inference in parralel gave better results. I was using 2 x E5-2660's.
Re: Reproducing GPT-2 in llm.c
#48Earlier quoted context omitted.
How big of a perf improvement would result from using the architectural tweaks that Llama3 and others have put in place since GPT-2?
My understanding and suspicion is mostly less than you think. Llama 3 architecture has the following changes on GPT-2: 1. delete the absolute positional encoding and replace with RoPE 2. delete all biases in all layers (in LayerNorms, they turn into RMSNorm) 3. GeLU -> SwiGLU non-linearity in the MLP 4. longer context length 5. architecture hyperparameter changes, e.g. slightly different aspect ratios And there was a…
Note llama's feed forward is a bit different too:
self.w2(F.silu(self.w1(x)) * self.w3(x))
https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...
Re: Reproducing GPT-2 in llm.c
#49[deleted]
Re: Reproducing GPT-2 in llm.c
#50Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!
> Keep in mind that here we trained for 10B tokens, while GPT-3 models were all trained for 300B tokens. [...] GPT-3 actually didn't change too much at all about the model (context size 1024 -> 2048, I think that's it?).
Andrej, based on that do you have a rough cost estimate for what it would take to train a GPT-3 Ada (350M)? Do you plan to get there with llm.c ?