Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU

arxiv.org

Post reply on HN
Print view View original

41–50 of 71 posts

Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU

#41

Post by stevemk14ebr »

kouteiheika wrote: post #22

This isn't really anything new; I've been doing something like this for quite a while, I just haven't bothered writing a paper. (: Probably anyone who would seriously tackle the problem of "how do I train a huge model on a tiny amount of VRAM?" would come up with something similar. However, most people in the field don't, because the actual practical utility of training huge models on a single GPU is quite low. (e.g…

As the saying goes, POC or GTFO

I invented faster than light travel, it was obvious, just didn't write a paper yet either :)

Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU

#43

Post by olliepro »

greenavocado wrote: post #11
olliepro wrote: post #4

This would likely only get used for small finetuning jobs. It’s too slow for the scale of pretraining.

So distribute copies of the model in RAM to multiple machines, have each machine update different parts of the model weights, and sync updates over the network

decentralized training makes a lot more sense when the required hardware isn't a $40K GPU...

Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU

#44

Post by kouteiheika »

gavinray wrote: post #40
kouteiheika wrote: post #26

Earlier quoted context omitted.

> You can only give it a try, but don't get your hopes high on a large context. You may or may not know this, but: when training off-the-shelf LLMs (i.e. ones which have a huge vocabulary) what consumes a huge amount of memory usage is calculating the cross-entropy loss (which gets worse the more tokens you stuff in your batch), so always use a fused cross-entropy kernel. For example, for a Gemma 2 model with 2B para…

I'd not heard of this before, quick search turned up this 2025 post which suggests "fused cross-entropy loss" kernel was integrated into PyTorch: https://pytorch.org/blog/peak-performance-minimized-memory/ > "The integration involves modifying the TransformerDecoder module in torchtune to bypass the linear layer computation, allowing the Liger Fused Linear Cross Entropy Loss to handle the forward projection weights.…

Yes.

Although this wasn't integrated into PyTorch itself (but to torchtune, which is a different thing). If you're writing your own training loop you need to use a third-party kernel, e.g. the Liger kernel mentioned in the article, or Cut Cross Entropy (which is much better than the Liger one, although IIRC it has a numeric bug in one of its kernels making the results very slightly off).

Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU

#45

Post by kouteiheika »

hirako2000 wrote: post #37
kouteiheika wrote: post #26

Earlier quoted context omitted.

> You can only give it a try, but don't get your hopes high on a large context. You may or may not know this, but: when training off-the-shelf LLMs (i.e. ones which have a huge vocabulary) what consumes a huge amount of memory usage is calculating the cross-entropy loss (which gets worse the more tokens you stuff in your batch), so always use a fused cross-entropy kernel. For example, for a Gemma 2 model with 2B para…

Activation would still require gigabytes for a few kb context. There are plenty of techniques to optimise. But the question is what can an rtx 3080 train before OOM. The answer is not that much. Can barely do quantized fine tuning. Even then, small context.

> Activation would still require gigabytes for a few kb context.

For that you use activation checkpointing, and you can also offload that to the CPU in a smart way to hide the latency. Although, yes, for long context training the activations do dominate the memory usage (and quantizing them degrades things more than just quantizing weights and/or optimizer states).

Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU

#46

Post by p_stuart82 »

nekusar wrote: post #33
bilekas wrote: post #31

> H200 GPU with 1.5TB host memory, While yes it's one GPU.. It's not exactly a slim one.

When the comparison is again 128 H100's , yeah, this is a crazy good upgrade. And you can rent H100's and H200s for not that much per hour.

$2-4/hr always sounds cheap until you multiply by wall clock and reruns

Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU

#47

Post by ur-whale »

Why is it no one ever talks about the one thing no one can get their hands on except the big labs ?

I'm talking about the training set.

Sure there are some open sets out there.

But my guess is they are nowhere near what OpenAI, Google and Anthropic are actually using.

Happy to be proven wrong.

Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU

#48

Post by jeremyjh »

nekusar wrote: post #33
bilekas wrote: post #31

> H200 GPU with 1.5TB host memory, While yes it's one GPU.. It's not exactly a slim one.

When the comparison is again 128 H100's , yeah, this is a crazy good upgrade. And you can rent H100's and H200s for not that much per hour.

Yes but they are getting only 341 tok/s. A 2.5 trillion run would take over 200 years.

Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU

#50

Post by logicallee »

internetguy wrote: post #3

> MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a…

Could I ask what you train your models to do? How do you generate the training data for it?
Post reply on HN