Training AI models might not need enormous data centres
1–10 of 59 posts
Re: Training AI models might not need enormous data centres
#2Re: Training AI models might not need enormous data centres
#3Related:
New Training Technique for Highly Efficient AI Methods (2 points, 5 hours ago) https://news.ycombinator.com/item?id=42690664
DiLoCo: Distributed Low-Communication Training of Language Models (46 points, 1 year ago, 14 comments) https://news.ycombinator.com/item?id=38549337
Re: Training AI models might not need enormous data centres
#4It’s talking about training 10b parameters “capable models” with less compute using ew techniques, but top models will always need more
Re: Training AI models might not need enormous data centres
#5Related: New Training Technique for Highly Efficient AI Methods (2 points, 5 hours ago) https://news.ycombinator.com/item?id=42690664 DiLoCo: Distributed Low-Communication Training of Language Models (46 points, 1 year ago, 14 comments) https://news.ycombinator.com/item?id=38549337
Thanks for the links, some interesting discussion there.
The second article you linked indicates there will still be intense bandwidth requirements during training, shipping around gradient differentials.
What has changed in the past year? Is this technique looking better, worse, or the same?
Re: Training AI models might not need enormous data centres
#6Open source public models trained on kosher data are substantially derisking the AI hype. It makes a lot of sense to push this approach as far as it can get. Its similar to SETI at home etc. but potentially with far more impact.
Re: Training AI models might not need enormous data centres
#7Open source public models trained on kosher data are substantially derisking the AI hype. It makes a lot of sense to push this approach as far as it can get. Its similar to SETI at home etc. but potentially with far more impact.
They could, would and should. But: Training a state of the art LLM costs millions in GPU, electricity alone. There is no "open" organization at this point that can cover this. Current "open source public models" are shared by big players like Meta to undermine the competition. And they only publish their weights, not the training data, training protocols, training code; meaning it's not reproducible, and questionable if the training data is kosher.
Re: Training AI models might not need enormous data centres
#8Open source public models trained on kosher data are substantially derisking the AI hype. It makes a lot of sense to push this approach as far as it can get. Its similar to SETI at home etc. but potentially with far more impact.
They could, would and should. But: Training a state of the art LLM costs millions in GPU, electricity alone. There is no "open" organization at this point that can cover this. Current "open source public models" are shared by big players like Meta to undermine the competition. And they only publish their weights, not the training data, training protocols, training code; meaning it's not reproducible, and questionable…
Doesn’t Deepseek somewhat counter this narrative?
Re: Training AI models might not need enormous data centres
#9It’s talking about training 10b parameters “capable models” with less compute using ew techniques, but top models will always need more
Wow, yeah a 10B parameter model is pretty tiny and 300 3-GPU clusters for $18M is not really cheap.
I guess enormous is in the eye of the beholder.
Re: Training AI models might not need enormous data centres
#10Open source public models trained on kosher data are substantially derisking the AI hype. It makes a lot of sense to push this approach as far as it can get. Its similar to SETI at home etc. but potentially with far more impact.
[deleted]