Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

gilesthomas.com

Post reply on HN
Print view View original

111–120 of 124 posts

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#111

Post by rvnx »

jbs789 wrote: post #33
rvnx wrote: post #9

Earlier quoted context omitted.

I think I should have replied as a totally separate comment. This is my mistake. It is nice that the author shared the results of his exercise / experiment. Just got sad as I was reminded (when the 100 USD were mentioned) that all this game is 90%+ about money and hardware rather than skills. That being said I really like the initiative of the author.

I understand the emotional aspect of feeling like it’s out of reach for you. Thing is, if you focus on your own skill development and apply it at even a small scale, very few people do that. Then you go for a job and guess what, the company has resources you can leverage. Then you do that, and ultimately you could be in a position to have the credibility to raise your own capital. Play the long game and do what you c…

Not at all. The majority with the current AI craze not really about credibility or skills. It's like a kitchen.

Take a genius chef but give him rotten ingredients. He sweats, he tries, but the meal is barely edible. That's the $100 exercise, but only experts recognize the talent behind.

Take an unskilled cook but give him A5 Wagyu and prepared truffles. The result tastes amazing to the average person who will claim the chef is great (the investors).

It's about access to capital and selling a story ('ex'-Googler doesn't make you competent), not skills.

Great chefs in dark alleys go unnoticed.

Mediocre tourist traps near the Eiffel Tower are fully booked.

Look at Inflection AI. Average results, yet massive funding. They have the "location" and the backing, so they win. It's not about who cooks better; it's about who owns the kitchen but who sells a dream that tomorrow the food will be better.

We don't talk about small funding, we talk about 1.3 billion USD, just for that specific example, yet a tourist trap (using name-dropping / reputation instead of talent)

Snake-oil is rewarded as much as, or even more than real talent; a lot of people cannot see the difference between a chef and the ingredients, this is what I think is sad.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#112

Post by gpjt »

muricula wrote: post #98
nico wrote: post #53

Has anyone done something like this but with apple silicon instead of a graphics card? Training a small LLM on an M2-M5?

I've played with something similar with my M1 using Apple's MLX framework. The problem is I'm compute bound. I've never managed to get my M1 Max's GPU to process more than ~7.8k tokens per second at bf16 precision, so to train a 112M parameter model on ~20 billion tokens I'd need to run the model training for ~30 days. One solution is to reduce the scope of the problem -- you can train on a smaller less diverse datas…

OP here -- with a 112M model you should be able to get something worth playing with using 2.24B tokens. The Chinchilla heuristic is tokens = 20 x parameters. Obviously you cam get a better result by grinding through more tokens, but it will be very slow progress. It's worth noting that Andrej Karpathy is using the 20x thing for his nanochat project.

I try to explain the Chinchilla paper in the post, but your favourite AI should be able to explain it well, and has the benefit that you can ask follow-up questions.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#113

Post by gpjt »

spi wrote: post #108
gpjt wrote: post #92

Earlier quoted context omitted.

OK, early indicators support both you and Gemini quite strongly re: batch size. On my (somewhat ad-hoc) test dataset, I get losses like this: * OpenAI medium weights: 3.231 * OpenAI small weights: 3.500 * My locally trained model, FineWeb Chinchilla, batch size 6: 3.944 * My locally trained model, FineWeb-Edu Chinchilla, batch size 6: 4.167 * My locally trained model, FineWeb-Edu double Chinchilla, batch size 6: 4.13…

Thanks, very nice to see these results! Certainly using GPUs with more RAM makes things simpler to scale. Gradient accumulation is as easy as adding a counter for number of steps and an "if counter % gradient_accumulation_steps:` around `optimizer.step()`, so that can also be tried simply on a single GPU / cheaper GPUs. But if you can just use 8xA100 and your pipeline parallizes well, you also get results (almost) 8…

Exactly! If I can get it down to an hour or two (seems very plausible on an 8x H200 with 160 GiB VRAM per GPU, though those are almost never available on Lambda Labs), I'll do the experiments with dropout and the other possible causes of issues, then see if I can bake that all into a new train on the RTX 3090 and confirm it repros there. Looks like I'll definitely need gradient accumulation there.

I assume the zero_grad would need to go in the same if block?

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#114

Post by alansaber »

spi wrote: post #109
alansaber wrote: post #55

Earlier quoted context omitted.

To caveat, smaller batch sizes are generally better for model stability, but we go bigger because it substantially speeds up training

Mmh not really. As OP shows, speed increases with larger batch size, but only initially, until the GPU has high enough utilization; then speed improvements flatten out (although you might get OOM before that and not "really" see the flat part). Using smaller batch size increases _noise_, so quite literally decreases stability. That might be good sometimes: in the limit case, if the batch is as large as your training…

Are you sure about the top-cap on batch size for speed? See https://arxiv.org/pdf/1904.00962

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#115

Post by DenisM »

cultofmetatron wrote: post #88
danielmarkbruce wrote: post #68

Earlier quoted context omitted.

One needs about 12 to 18 hours of linear algebra to work though the papers, not 12 to 18 months. The vast majority of stuff in AI/ML papers is just "we tried X and it worked!".

for anyone looking to get into it, mathacademy has a full zero to everythign you need pathway that you can follow to mastery https://mathacademy.com/courses/mathematics-for-machine-lear...

There is no mention of llm there?

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#116

Post by gpjt »

gpjt wrote: post #113
spi wrote: post #108

Earlier quoted context omitted.

Thanks, very nice to see these results! Certainly using GPUs with more RAM makes things simpler to scale. Gradient accumulation is as easy as adding a counter for number of steps and an "if counter % gradient_accumulation_steps:` around `optimizer.step()`, so that can also be tried simply on a single GPU / cheaper GPUs. But if you can just use 8xA100 and your pipeline parallizes well, you also get results (almost) 8…

Exactly! If I can get it down to an hour or two (seems very plausible on an 8x H200 with 160 GiB VRAM per GPU, though those are almost never available on Lambda Labs), I'll do the experiments with dropout and the other possible causes of issues, then see if I can bake that all into a new train on the RTX 3090 and confirm it repros there. Looks like I'll definitely need gradient accumulation there. I assume the zero_g…

Hmm, interesting. With a batch size of 512 (8x B200s with 160 GiB each) I get worse results! Maybe there's a sweet spot somewhere in between.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#117

Post by cultofmetatron »

DenisM wrote: post #115
cultofmetatron wrote: post #88

Earlier quoted context omitted.

for anyone looking to get into it, mathacademy has a full zero to everythign you need pathway that you can follow to mastery https://mathacademy.com/courses/mathematics-for-machine-lear...

There is no mention of llm there?

if you want to use llms, just download one and play with it. if you want to understand llms enough to push research forward, learn the underlying math

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#118

Post by spi »

alansaber wrote: post #114
spi wrote: post #109

Earlier quoted context omitted.

Mmh not really. As OP shows, speed increases with larger batch size, but only initially, until the GPU has high enough utilization; then speed improvements flatten out (although you might get OOM before that and not "really" see the flat part). Using smaller batch size increases _noise_, so quite literally decreases stability. That might be good sometimes: in the limit case, if the batch is as large as your training…

Are you sure about the top-cap on batch size for speed? See https://arxiv.org/pdf/1904.00962

Sorry I just opened that file now, and browsed through it very quickly, but my eye fell on the excerpt: ``` However, we did not observe any speedup by increasing the batch size from 65536 to 131072 for the first stage, thus, we restrict the batch size to 65536 for this stage. ``` which I think is more or less my point: increasing batch size essentially always helps, but the speedup reduces the more you push the batch size. Provided that your dataset is large enough, more batch size will always make you run a bit faster without sacrificing accuracy, but the speedup will be less and less as you increase the batch size, until you are anyway maxing out the power of your GPU and you can't see any measurable speedup anymore.

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#119

Post by spi »

gpjt wrote: post #116
gpjt wrote: post #113

Earlier quoted context omitted.

Exactly! If I can get it down to an hour or two (seems very plausible on an 8x H200 with 160 GiB VRAM per GPU, though those are almost never available on Lambda Labs), I'll do the experiments with dropout and the other possible causes of issues, then see if I can bake that all into a new train on the RTX 3090 and confirm it repros there. Looks like I'll definitely need gradient accumulation there. I assume the zero_g…

Hmm, interesting. With a batch size of 512 (8x B200s with 160 GiB each) I get worse results! Maybe there's a sweet spot somewhere in between.

Sorry came a bit late to this reply. Interesting, well, nobody says it's a monotonic function :-) in the limit of _very_ large batches you of course are worse off, because you take a very large amount of computation before taking a single step, so if you stop after a fixed amount of time your model just didn't have the time to learn properly. So certainly there is a sweet spot somewhere.

I suppose, the real "function" is a bit more complicated because (1) If you put 2x more data through the same GPU with large enough memory, it will take less than 2x the time to compute (but certainly not 1x). (2) At some point, empirically, increasing batch size makes it _worse_ even if you ignore the additional runtime cost (i.e. stop after n gradient update steps, and not x seconds). To my knowledge, the accepted reason for that fact is that a bit of noise helps in regularizing learning, because overly smooth learning curves end up stagnating in local loss minima more easily. In truth, I think nobody exactly understand how deep learning models work :-)

And to your other question - sorry again for the late answer. Yes, `optimizer.zero_grad()` should always be called directly after `optimizer.step()`, therefore with gradient accumulation once every `n` steps (otherwise, you'd be zeroing out the gradients, so just throwing away all the compute you did in previous steps).

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#120

Post by gpjt »

spi wrote: post #119
gpjt wrote: post #116

Earlier quoted context omitted.

Hmm, interesting. With a batch size of 512 (8x B200s with 160 GiB each) I get worse results! Maybe there's a sweet spot somewhere in between.

Sorry came a bit late to this reply. Interesting, well, nobody says it's a monotonic function :-) in the limit of _very_ large batches you of course are worse off, because you take a very large amount of computation before taking a single step, so if you stop after a fixed amount of time your model just didn't have the time to learn properly. So certainly there is a sweet spot somewhere. I suppose, the real "function…

Thanks re: gradient accumulation, I'm glad to hear my intuition was right!

As part of the upcoming post I'm running the DDP train on A100s with 40 GiB and 80 GiB, H100s with 80 GiB, and B200s with 160 GiB, so I'll have at least three loss vs. batch size points to plot. So that might be interesting.

I guess a full test would be to train at various batch sizes on the 160 GiB machine and plot the resulting loss. That would be very expensive as a hobby project (the bs=64 train cost a bit more than $40 excluding overhead) so I won't do it.

But perhaps a shorter train would still be of value? That is, train for 300M tokens for a tenth of the cost and see where the loss landed? The problem with that would be if the impact of batch sizes varied with the length of the train, eg. if batch size 64 was better than 512 for short trains but weaker at longer ones.

Post reply on HN