41–50 of 59 posts
It’s talking about training 10b parameters “capable models” with less compute using ew techniques, but top models will always need more
It’s talking about training 10b parameters “capable models” with less compute using ew techniques, but top models will always need more
This approach likely scales to top models. There was a thought for a while among researchers that scaling up Hogwild / LocalSGD to more parameters would be less convergent, but it seems to be that things actually converge a bit better with larger models, especially if a MoE split is used.
Federated learning breaks the barrier to entry and expands the ecosystem allowing more participants to share compute and/or datasets for small players to train models.
DiLoCo introduced by Douillard minimizes communication overhead by averaging weight updates. What this article misses though is that despite this, each GPU in the distributed cluster still needs to have enough VRAM to load the entire copy of the model to complete the training process. That's where DisTrO comes in which even reduces further the inter-GPU communication using a decoupling technique (DeMo) that only shares the fast moving parts of the optimizer across the GPU cluster.
>And what if the costs could drop further still? The dream for developers pursuing truly decentralised ai is to drop the need for purpose-built training chips entirely. Measured in teraflops, a count of how many operations a chip can do in a second, one of Nvidia’s most capable chips is roughly as powerful as 300 or so top-end iPhones. But there are a lot more iPhones in the world than gpus. What if they (and other consumer computers) could all be put to work, churning through training runs while their owners sleep?"
This aligns with DisTrO techniques because, according to them it could also allow consumer devices like Desktop Gaming PCs to join the compute cluster and share workloads. Besides there's also an open-source implementation called exo that allows models to be split among idle local devices but it's only limited to inference.
Again might still be relevant since in the article it mentions that DiLoCo was able to make the model respond better when faced with instruction prompts or reasoning questions never encountered during pre-training. And Arthur seems to think test-time training will make his approach become the norm.
sources: DisTrO: https://github.com/NousResearch/DisTrO DeMo: https://arxiv.org/pdf/2411.19870 Exo: https://github.com/exo-explore/exo
Related: New Training Technique for Highly Efficient AI Methods (2 points, 5 hours ago) https://news.ycombinator.com/item?id=42690664 DiLoCo: Distributed Low-Communication Training of Language Models (46 points, 1 year ago, 14 comments) https://news.ycombinator.com/item?id=38549337
Thanks for the links, some interesting discussion there. The second article you linked indicates there will still be intense bandwidth requirements during training, shipping around gradient differentials. What has changed in the past year? Is this technique looking better, worse, or the same?
There was a distributed Protein Folding project a couple decades ago.
I remember there was even Protein folding apps that could run on game consoles when not playing games.
But maybe Protein Folding code is more Parallelizable across machines, than AI models.
You could consider a LLM as a very lossy compression artifact. Where they took terabytes of input data, and ended up with model under the 100 gigabytes. It is quite remarkable what such a model can do, even fabricating new output that was not in the input data. However, in my naïvety, I wonder whether vastly simpler algorithms could be used to end up with similar results. Regular compression techniques work with spee…
The remarkable thing about this compression method is that stochastic gradient descent for some reason creates algorithms in the network. Not Turing-complete algorithms, of course, but algorithms nevertheless. An LLM trained on the addition and multiplication data develops circuits for addition and multiplication[1]. It stands to reason that LLM trained on human-produced data develop algorithms that try to approximat…
Earlier quoted context omitted.
I think it's important to remember that we know neural networks can be trained to a very useful state from scratch for 24 GJ: This is 25 W for 30 years (or 7000 kWh, or a good half ton of diesel fuel), which is what a human brain consumes until adulthood. Even though our artificial training efficiency is worse now, likely to stay worse because we want to trade efficiency for faster training, and because we want to cr…
This misses that evolution has been pre-training the human cognitive architecture - brain, limbic system, sympathetic and parasympathetic nervous systems, coevolved viral and bacterial ecosystems - for millions of years. We're not a tabula rasa training at birth to perfectly fit whatever set of training data we're presented. Far from it. Human learning is more akin to RAG, or test time training - specialising a heavi…
Yes. But that is not part of the training cost; this is basically the equivalent to figuring out a suitable artificial neural net architecture and hyperparameter tuning in general. That is not energy cost that you pay per training run, but fixed cost overhead instead.
You raise a good point that when doing artificial training, the "environment" has to be provisioned as well (i.e. feeding audio/visual/text input in some way to do the training), but here I would argue that in energy terms, that is a rather small overhead (less than an order of magnitude) because our digital information storage/transmission capabilities are frankly insane compared to a human already (and reasonably efficient as well).
Open source public models trained on kosher data are substantially derisking the AI hype. It makes a lot of sense to push this approach as far as it can get. Its similar to SETI at home etc. but potentially with far more impact.
what's "kosher data"? Never heard of that before
Earlier quoted context omitted.
> This misses that evolution has been pre-training the human cognitive architecture - brain, limbic system, sympathetic and parasympathetic nervous systems, coevolved viral and bacterial ecosystems - for millions of years. We're not a tabula rasa training at birth to perfectly fit whatever set of training data we're presented. Far from it. Human learning is more akin to RAG Yes, but. The human genome isn't that big (…
You've missed the fact that sequencing our genome isn't gathering all the information required. To duplicate a human in computational space - say to create some accelerated AI simulation, you'd need to sequence a complete Telomere-to-Telomere genome (something achieved for the first time only last year!), complete Centromere sequencing (not yet achieved). You'd also need to 'sequence' or somehow encode the epigenome…
Knowing the weights without knowing the full graph of the model they're used in, just the endpoints.
There's a lot of valid stuff in what you say, I am aware I'm glossing over a lot of challenges to get a copy of a human — to what extent is e.g. the microbiome even contributing to our intelligence, vs. being several hundred different parasites that share a lot of DNA with each other and which happen to accidentally also sometimes give us useful extras? It's hard work telling which is which — but my claim is that the nature and scope of such work itself still allows us to say, as per one of the parent comments:
> I think it's important to remember that we know neural networks can be trained to a very useful state from scratch for 24 GJ: This is 25 W for 30 years (or 7000 kWh, or a good half ton of diesel fuel), which is what a human brain consumes until adulthood.
If this were a 100m sprint, then I would agree with you essentially saying that we don't even know which country the starting blocks are in, but I am still saying that despite that we know the destination can be reached from the starting blocks in 10 seconds.
This bottleneck right here is why Open Source is presented with a golden plate opportunity to lead the training of cutting edge models. Federated learning breaks the barrier to entry and expands the ecosystem allowing more participants to share compute and/or datasets for small players to train models. DiLoCo introduced by Douillard minimizes communication overhead by averaging weight updates. What this article misse…
That's not exactly accurate. In the data parallel side of techniques, the Distributed Data Parallel (DDP) approach does require a fully copy of the model on each GPU. However there's also Fully Sharded Data Parallel (FSDP) which does not.
Similarly things like tensor parallelism (TP) split the model over GPUs, to the point where full layers are never in a single GPU anymore.
Combining multiple of the above is how huge foundation models are trained. Meta used 4d parallelism (FSDP + TP and pipeline/context parallelism) to train llama 405b.