[dead]
MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
61–70 of 71 posts
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#62sounds very similar to https://docs.pytorch.org/docs/stable/distributed.fsdp.fully_... i wonder how much this could be replicated using only this pytorch primitive
Check out Fig. 6 in this paper, it shows the comparison between the proposed method and pytorch native FSDP offload method.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#63This is a fantastic step toward democratizing large model training. Making 100B+ parameter training accessible on a single GPU could open the door to a lot more independent research. Really impressive work!
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#64Why is it no one ever talks about the one thing no one can get their hands on except the big labs ? I'm talking about the training set. Sure there are some open sets out there. But my guess is they are nowhere near what OpenAI, Google and Anthropic are actually using. Happy to be proven wrong.
I think OpenAI and Anthropic just downloaded the same torrents from Anna's Archive that anyone else can. But it's only OK when they do it. The rest of us get nastygrams from law offices. Anthropic actually had to cough up some bucks, for that matter. At that point, a lot depends on the quality of the preprocessing applied to the raw text dumps. It is reportedly not that trivial to go from DumpOfSketchyRussianPirateSi…
> I think OpenAI and Anthropic just downloaded the same torrents from Anna's Archive
I think you're underestimating how much chat conversation data they've gathered at this point, and how much of it is part of the training set.
None of that is available to anyone who wants to train a frontier model.
And when it comes to Google ... the hoard of data they're sitting on goes back to what? 1998? They've basically got a digital record of what happened since the birth of the internet.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#65[dead]
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#66> H200 GPU with 1.5TB host memory, While yes it's one GPU.. It's not exactly a slim one.
even if it scales down, the host-to-gpu memory ratio is around 10 to 1 (1.5 TB vs 141 GB) [1].
might need some odd system builds, improbable with current pricing, to replicate or better the ratio. such as a ~256 GB system for a 4090 (with 24 GB VRAM).
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#67Earlier quoted context omitted.
To make the most of these architectures I think the key is essentially moving more of the knowledge/capabilities out of the "weights" and into the complimentary parts of the system in a way that's proportionate to the capabilities of the hardware. In the past couple months there's been a kind of explosion in small-models that are occupying a niche in this kind of AI-transcoding space. What I'm hoping we're right on t…
You are on the right track. Check out the Semiotic-Reflexive Transformer (SRT) here. https://open.substack.com/pub/sublius/p/the-semiotic-reflexi...
Empirically proved a new science. Still gets downvoted xD.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#68> H200 GPU with 1.5TB host memory, While yes it's one GPU.. It's not exactly a slim one.
That’s a pretty non-standard H200 configuration. In the regular HGX configurations, a node with 8xH200 has that much CPU DRAM. That makes the title of the paper somewhat arguable imo.
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#69> MegaTrain stores parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state This is pretty awesome. The only compute I have at home is an RTX 3080 with 10 GB of VRAM, so I struggle with training larger models (>40M, 50M params). I get OOM errors and have to optimize a…
To make the most of these architectures I think the key is essentially moving more of the knowledge/capabilities out of the "weights" and into the complimentary parts of the system in a way that's proportionate to the capabilities of the hardware. In the past couple months there's been a kind of explosion in small-models that are occupying a niche in this kind of AI-transcoding space. What I'm hoping we're right on t…
> To make the most of these architectures I think the key is essentially moving more of the knowledge/capabilities out of the "weights" and into the complimentary parts of the system in a way that's proportionate to the capabilities of the hardware
I think that's only possible to limited extent. Learnt skills (RL in context of an LLM?) need to be in the weights of the model since this reflects the model's "personalized" learning of the behavioral feedback loop. Declarative knowledge (facts) can be loaded at runtime (RAG).
Re: MegaTrain: Full Precision Training of 100B+ Parameter LLMs on a Single GPU
#70Having just started to dabble with training LLMs, it seems training a model if you have a training and validation data set is fairly trivial. Creating a good and sufficiently large training and validation data set seems to be the hard part.
Sourcing, cleaning, curating, labeling, generating and quality controlling training data is hard and a lot of work, at least has been for the projects I've dabbled with.