> H200 GPU with 1.5TB host memory,
While yes it's one GPU.. It's not exactly a slim one.
MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
31–40 of 71 posts
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#32Seems similar to Microsoft DeepSpeed.
The compare against “DeepSpeed ZeRO-3” apparently.
FWIW Zero-3 refers to a common strategy for sharding model components across GPUs (commonly called FSDP-2, Full Sharded Data Parallel). The "3" is the level of sharding (how much stuff to distribute across GPUs, e.g. just weights, versus optimizer state as well, etc.)
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#33> H200 GPU with 1.5TB host memory, While yes it's one GPU.. It's not exactly a slim one.
When the comparison is again 128 H100's , yeah, this is a crazy good upgrade.
And you can rent H100's and H200s for not that much per hour.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#34> H200 GPU with 1.5TB host memory, While yes it's one GPU.. It's not exactly a slim one.
>cries in RTX 3060
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#35This isn't really anything new; I've been doing something like this for quite a while, I just haven't bothered writing a paper. (: Probably anyone who would seriously tackle the problem of "how do I train a huge model on a tiny amount of VRAM?" would come up with something similar. However, most people in the field don't, because the actual practical utility of training huge models on a single GPU is quite low. (e.g…
Can you take the time to write your methods? I’d be interested in reading it
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#36This isn't really anything new; I've been doing something like this for quite a while, I just haven't bothered writing a paper. (: Probably anyone who would seriously tackle the problem of "how do I train a huge model on a tiny amount of VRAM?" would come up with something similar. However, most people in the field don't, because the actual practical utility of training huge models on a single GPU is quite low. (e.g…
341 is two orders of magnitude faster than your 1 tok/s so it doesn’t seem like their stuff is all that obvious. I also have no baseline for training to know if 341tok/s is slow but it seems speedy for a 3090.
:) Coffee is good
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#37Earlier quoted context omitted.
The claims of the article assumes far more compute and far more VRAM..while the trick enables less back and forth, they don't eliminate it. I doubt you meant 50M. Rather 50B? You can only give it a try, but don't get your hopes high on a large context. If their technique works I would guess 8096k context limits would still OOM. 2048 maybe. I'm extrapolating based on my experiment without this paper's trick to leverag…
> You can only give it a try, but don't get your hopes high on a large context. You may or may not know this, but: when training off-the-shelf LLMs (i.e. ones which have a huge vocabulary) what consumes a huge amount of memory usage is calculating the cross-entropy loss (which gets worse the more tokens you stuff in your batch), so always use a fused cross-entropy kernel. For example, for a Gemma 2 model with 2B para…
Activation would still require gigabytes for a few kb context.
There are plenty of techniques to optimise. But the question is what can an rtx 3080 train before OOM. The answer is not that much.
Can barely do quantized fine tuning. Even then, small context.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#38This isn't really anything new; I've been doing something like this for quite a while, I just haven't bothered writing a paper. (: Probably anyone who would seriously tackle the problem of "how do I train a huge model on a tiny amount of VRAM?" would come up with something similar. However, most people in the field don't, because the actual practical utility of training huge models on a single GPU is quite low. (e.g…
341 is two orders of magnitude faster than your 1 tok/s so it doesn’t seem like their stuff is all that obvious. I also have no baseline for training to know if 341tok/s is slow but it seems speedy for a 3090.
OOM is log10
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#39I’m curious how this technique works, or not, with unified memory architectures such as Apple’s M series. It seems like it’s relying on using overlapping processes to help speed things up, but I would assume that having everything unified in main memory such that you don’t have to transfer everything back and forth to the GPU would also have some advantages. Can someone wiser explain this to me?
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#40Earlier quoted context omitted.
The claims of the article assumes far more compute and far more VRAM..while the trick enables less back and forth, they don't eliminate it. I doubt you meant 50M. Rather 50B? You can only give it a try, but don't get your hopes high on a large context. If their technique works I would guess 8096k context limits would still OOM. 2048 maybe. I'm extrapolating based on my experiment without this paper's trick to leverag…
> You can only give it a try, but don't get your hopes high on a large context. You may or may not know this, but: when training off-the-shelf LLMs (i.e. ones which have a huge vocabulary) what consumes a huge amount of memory usage is calculating the cross-entropy loss (which gets worse the more tokens you stuff in your batch), so always use a fused cross-entropy kernel. For example, for a Gemma 2 model with 2B para…
I'd not heard of this before, quick search turned up this 2025 post which suggests "fused cross-entropy loss" kernel was integrated into PyTorch:
https://pytorch.org/blog/peak-performance-minimized-memory/
> "The integration involves modifying the TransformerDecoder module in torchtune to bypass the linear layer computation, allowing the Liger Fused Linear Cross Entropy Loss to handle the forward projection weights. "