LLM from scratch, part 28 – training a base model from scratch on an RTX 3090 • Hacker News Board

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#121

Post by tomrod » Fri, Dec 12, 2025, 3:25 PM UTC

whimsicalism wrote: post #87

miki123211 wrote: post #74

Earlier quoted context omitted.

You can understand 95+% of current LLM / neural network tech if you know what matrices are (on the "2d array" level, not the deeper lin alg intuition level), and if you know how to multiply them (and have an intuitive understanding why a matrix is a mapping between latent spaces and how a matrix can be treated as a list of vectors). Very basic matrix / tensor calculus comes in useful, but that's not really part of li…

> There are places where things like eigenvectors / eigenvalues or svd come into play, but those are pretty rare and not part of modern architectures (tbh, I still don't really have a good intuition for them) This stuff is part of modern optimizers. You can often view a lot of optimizers as doing something similar to what is called mirror/'spectral descent.'

Indeed. "Spectral" describes the collection of eigenvalues!

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#122

Post by tomrod » Fri, Dec 12, 2025, 3:26 PM UTC

cultofmetatron wrote: post #117

DenisM wrote: post #115

Earlier quoted context omitted.

There is no mention of llm there?

if you want to use llms, just download one and play with it. if you want to understand llms enough to push research forward, learn the underlying math

100%

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#123

Post by spi » Sat, Dec 13, 2025, 8:40 PM UTC

gpjt wrote: post #120

spi wrote: post #119

Earlier quoted context omitted.

Sorry came a bit late to this reply. Interesting, well, nobody says it's a monotonic function :-) in the limit of _very_ large batches you of course are worse off, because you take a very large amount of computation before taking a single step, so if you stop after a fixed amount of time your model just didn't have the time to learn properly. So certainly there is a sweet spot somewhere. I suppose, the real "function…

Thanks re: gradient accumulation, I'm glad to hear my intuition was right! As part of the upcoming post I'm running the DDP train on A100s with 40 GiB and 80 GiB, H100s with 80 GiB, and B200s with 160 GiB, so I'll have at least three loss vs. batch size points to plot. So that might be interesting. I guess a full test would be to train at various batch sizes on the 160 GiB machine and plot the resulting loss. That wo…

Yes exactly, I fear that shortening the training time would skew the results. In the very short term, smaller batch size is typically better just because you need a certain amount of gradient updates to move away from the original random, hence pretty terrible, weight distribution. Larger batch size gives a steadier, but slower, convergence, so it's hard to say for sure what is better for a given compute budget.

I'm definitely _not_ encouraging you on spending more money on a side topic just for the sake of optimizing this one parameter, there will always be another parameter after that that you'll feel an urge to optimize :-) I'd say it's already a pretty neat result to have come to a very close score to the original GPT2 training starting from scratch!

P.S. If you want to push it a bit further, rather than optimizing parameters for this model, last week at EurIPS I heard that a current "very good" modern repo to start from in order to train a good LLM is this: https://github.com/Niccolo-Ajroldi/plainLM. I haven't investigated this exactly (I'm not working on LLM), but it might be interesting to you for a sample run. The (N)EurIPS paper that was discussed at the conference claimed that the only important change to do was to modify the hyperparameters of the Adam optimizer, setting beta1=beta2=0.95 for example (the default values are beta1=0.9 and beta2=0.999 which are apparently outdated).

Re: LLM from scratch, part 28 – training a base model from scratch on an RTX 3090

#124

Post by gpjt » Wed, Dec 17, 2025, 9:28 PM UTC

spi wrote: post #123

gpjt wrote: post #120

Earlier quoted context omitted.

Thanks re: gradient accumulation, I'm glad to hear my intuition was right! As part of the upcoming post I'm running the DDP train on A100s with 40 GiB and 80 GiB, H100s with 80 GiB, and B200s with 160 GiB, so I'll have at least three loss vs. batch size points to plot. So that might be interesting. I guess a full test would be to train at various batch sizes on the 160 GiB machine and plot the resulting loss. That wo…

Yes exactly, I fear that shortening the training time would skew the results. In the very short term, smaller batch size is typically better just because you need a certain amount of gradient updates to move away from the original random, hence pretty terrible, weight distribution. Larger batch size gives a steadier, but slower, convergence, so it's hard to say for sure what is better for a given compute budget. I'm…

Awesome, thanks! I'm still doing trains on the big machines right now (hopefully will write up over xmas) but I think once I've worked out the sweet spot for memgatokens per dollar for this model, it's time to start tweaking the other controls -- LR and cosine variation of it, as you said, and also dropout, bias, weight tying, and definitely gradient clipping (which should at least get better bang for the buck from time/$ spent). I'll leave it to Google to follow up Chinchilla with a "best batch size across a thousand trained models" paper ;-)