Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DiLoCo: Distributed Low-Communication Training of Language Models

arxiv.org

Post reply on HN
Print view View original

11–15 of 15 posts

Re: DiLoCo: Distributed Low-Communication Training of Language Models

#11

Post by dinobones »

vessenes wrote: post #8

This is really cool, and I expect groups like Eleuther will want to integrate it into their workflows. That said, there is a fundamental rule here -- gradients must, at some point, be shared. This paper says they don't have to be shared every step. Which is really great. But, gradient sizes are on the order of magnitude of the network being trained. To get this useful for indie LLM training, I'd guess that you'll wan…

Quick question: What is a "gradient" in this context? Is it a file? Is it some state that is stored somewhere?

My understanding of ML is, I made what is basically a "Hot Dog or Not Hot Dog" image classifier and know what a neural net is.

The gradient for that simple neural net, was just found by running Adam an optimizer on the current batch, and then updating the model weights. So by "gradient" do you mean model weights?

Re: DiLoCo: Distributed Low-Communication Training of Language Models

#12

Post by techwizrd »

There are a lot of distributed data parallel and federated learning (FL) algorithms that could be applied to training LLMs, and there have been several papers that tackle applying these. I don't think I've seen 500 steps in the FL literature (little eye opening to be honest), and I don't think I've tested more than 50 steps between communication rounds personally. I'd be interested in testing other algorithms, partial client participation, hierarchical federation approaches, error-feedback, and so on.

Re: DiLoCo: Distributed Low-Communication Training of Language Models

#13

Post by theendisney2 »

lucubratory wrote: post #2

So we can SETI@Home or Folding@Home for large language models, now? Not sure how small the minimum size of the compute cluster can be. If it's still out of consumer reach then this would either be just an intermediate research step, or a way for small-er (but still professional/well-resourced) labs to collaborate together. I'm not sure the latter would be helpful, as if they wanted to collaborate together they could…

Someone onhere argued this is entirely possible and that results can be merged later on. It seems facinating. Perhaps it is not conpute each node has to contribute but just bandwidth?

Re: DiLoCo: Distributed Low-Communication Training of Language Models

#14

Post by lumost »

dinobones wrote: post #11
vessenes wrote: post #8

This is really cool, and I expect groups like Eleuther will want to integrate it into their workflows. That said, there is a fundamental rule here -- gradients must, at some point, be shared. This paper says they don't have to be shared every step. Which is really great. But, gradient sizes are on the order of magnitude of the network being trained. To get this useful for indie LLM training, I'd guess that you'll wan…

Quick question: What is a "gradient" in this context? Is it a file? Is it some state that is stored somewhere? My understanding of ML is, I made what is basically a "Hot Dog or Not Hot Dog" image classifier and know what a neural net is. The gradient for that simple neural net, was just found by running Adam an optimizer on the current batch, and then updating the model weights. So by "gradient" do you mean model wei…

The gradient is the direction and magnitude of change for each model weight. The optimizer determines how to adjust the weight based on the gradient.

Re: DiLoCo: Distributed Low-Communication Training of Language Models

#15

Post by UncleOxidant »

dinobones wrote: post #11
vessenes wrote: post #8

This is really cool, and I expect groups like Eleuther will want to integrate it into their workflows. That said, there is a fundamental rule here -- gradients must, at some point, be shared. This paper says they don't have to be shared every step. Which is really great. But, gradient sizes are on the order of magnitude of the network being trained. To get this useful for indie LLM training, I'd guess that you'll wan…

Quick question: What is a "gradient" in this context? Is it a file? Is it some state that is stored somewhere? My understanding of ML is, I made what is basically a "Hot Dog or Not Hot Dog" image classifier and know what a neural net is. The gradient for that simple neural net, was just found by running Adam an optimizer on the current batch, and then updating the model weights. So by "gradient" do you mean model wei…

No, it's an adjustment to model weights that is made during training. Given some input and some expected value there will be some delta and that is used to calculate the gradient - these networks are essentially being trained by gradient descent. As for the size of that data that has to be shared it's going to depend on network size and what kind of representation you're using for the weights - probably bfloat16 these days, but we're certainly seeing a lot of 4 bit representations now.
Post reply on HN