How long would it actually take to train a 120B model on an H200? What if you have 8?
[deleted]
MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
51–60 of 71 posts
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#52> MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a…
> This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a lot. I'm on the same GPU, its intimidating to me if I even want to bother training anything at all. Do you mind sharing what kind of training you've done with that GPU? :)
Make sure you are running adaptive cooling (or just bump them up) on the top fans in your case. Also, ensure that you are undervolting appropriately using something like MSI Afterburner or GreenWithEnvy.
If you don't, you could easily toast your RAM -- especially under BF16.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#53I’m curious how this technique works, or not, with unified memory architectures such as Apple’s M series. It seems like it’s relying on using overlapping processes to help speed things up, but I would assume that having everything unified in main memory such that you don’t have to transfer everything back and forth to the GPU would also have some advantages. Can someone wiser explain this to me?
For FP16-native training of 100B+ models, you will probably still be offloading to swap unless you've got a $150,000 RDMA Mac Studio cluster. The workload would be deeply compute-constrained if you could fit it in-memory anyways.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#54Why is it no one ever talks about the one thing no one can get their hands on except the big labs ? I'm talking about the training set. Sure there are some open sets out there. But my guess is they are nowhere near what OpenAI, Google and Anthropic are actually using. Happy to be proven wrong.
I think OpenAI and Anthropic just downloaded the same torrents from Anna's Archive that anyone else can. But it's only OK when they do it. The rest of us get nastygrams from law offices. Anthropic actually had to cough up some bucks, for that matter.
At that point, a lot depends on the quality of the preprocessing applied to the raw text dumps. It is reportedly not that trivial to go from DumpOfSketchyRussianPirateSite.zip to a data set suitable for ingestion during pretraining. A few bad chunks of data can apparently do more harm than one would expect.
AFAIK Google scans almost everything in print as part of the Google Books initiative, so they may have been able to skip the torrenting step.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#55[dead]
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#56> MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a…
Anything that can run on a AMD395+ w/128GB or whatever the apple equivelent would break things wide open. Training a model on my frameworks of choice or our business info would be awesome.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#57I'm most likely wrong but large language models are literally just stealing....everything
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#58> MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a…
The claims of the article assumes far more compute and far more VRAM..while the trick enables less back and forth, they don't eliminate it. I doubt you meant 50M. Rather 50B? You can only give it a try, but don't get your hopes high on a large context. If their technique works I would guess 8096k context limits would still OOM. 2048 maybe. I'm extrapolating based on my experiment without this paper's trick to leverag…
Maybe they're not talking about LLMs? Training a 9M parameter YOLO can take over 20GB of VRAM if you use a large image size.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#59> MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a…
To make the most of these architectures I think the key is essentially moving more of the knowledge/capabilities out of the "weights" and into the complimentary parts of the system in a way that's proportionate to the capabilities of the hardware. In the past couple months there's been a kind of explosion in small-models that are occupying a niche in this kind of AI-transcoding space. What I'm hoping we're right on t…
That's interesting. So you want to train language, linguistic reasoning, and tool use, but otherwise strip out all knowledge in lieu of a massive context? Just grade they model on how well it can access local information, perhaps also run tools?
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#60[dead]