Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training AI models might not need enormous data centres

economist.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Training AI models might not need enormous data centres

#51

Post by HPsquared »

htrp wrote: post #39
mr_toad wrote: post #37

Earlier quoted context omitted.

Data parallel training is not the only approach. Sometimes the model itself needs to be distributed across multiple GPU. https://www.microsoft.com/en-us/research/blog/zero-deepspeed... The communications overhead of doing this over the internet might be unworkable though.

or if the internet became significantly faster fiber connections

A single GPU has memory bandwidth around 1000 GB/s ... that's a lot of fiber! (EDIT: although the PCIE interconnect isn't as fast, of course. NVLink is pretty fast though which is the sort of thing you'd be using in a large system)

Re: Training AI models might not need enormous data centres

#52

Post by brookst »

htrp wrote: post #39
mr_toad wrote: post #37

Earlier quoted context omitted.

Data parallel training is not the only approach. Sometimes the model itself needs to be distributed across multiple GPU. https://www.microsoft.com/en-us/research/blog/zero-deepspeed... The communications overhead of doing this over the internet might be unworkable though.

or if the internet became significantly faster fiber connections

Latency still matters a lot…

Re: Training AI models might not need enormous data centres

#53

Post by brookst »

ben_w wrote: post #28
openrisk wrote: post #6

Open source public models trained on kosher data are substantially derisking the AI hype. It makes a lot of sense to push this approach as far as it can get. Its similar to SETI at home etc. but potentially with far more impact.

I don't see how it really helps? We could pass laws requiring models to demonstrate their training sets irregardless of how the training is distributed; and conversely if this is a community-led project, those also have copyright issues to deal with (wikipedia for example). I suspect there's also a problem in that, e.g. ten million student essays about different pages of Harry Potter can each in isolation be justifie…

I think you’re doing away with the fairness exception for criticism?

Copyright is intended to reward investment in creative works by giving sole license to distribute. It is not intended to create a monopoly on knowledge about the work.

If I can ask an LLM (or person!) “what’s the first sentence in Harry Potter?” And then “what’s the second sentence?” and so on, that does not mean they are distributing the work in competition with the rights holders.

We have gone way overboard with IP protections. The purpose of copyright is served when Rowling buys her 10th mansion. We do not need to further expand copyright to make it illegal to learn from a work or to remember it after reading.

Re: Training AI models might not need enormous data centres

#54

Post by brookst »

falcor84 wrote: post #48
netdevphoenix wrote: post #30

Earlier quoted context omitted.

what's "kosher data"? Never heard of that before

I haven't heard of it either, but can only assume that it requires that data regarding dairy and meat MUST be processed on separate machines.

LLMs unable to reason about cheeseburgers…

Re: Training AI models might not need enormous data centres

#55

Post by ben_w »

brookst wrote: post #53
ben_w wrote: post #28

Earlier quoted context omitted.

I don't see how it really helps? We could pass laws requiring models to demonstrate their training sets irregardless of how the training is distributed; and conversely if this is a community-led project, those also have copyright issues to deal with (wikipedia for example). I suspect there's also a problem in that, e.g. ten million student essays about different pages of Harry Potter can each in isolation be justifie…

I think you’re doing away with the fairness exception for criticism? Copyright is intended to reward investment in creative works by giving sole license to distribute. It is not intended to create a monopoly on knowledge about the work. If I can ask an LLM (or person!) “what’s the first sentence in Harry Potter?” And then “what’s the second sentence?” and so on, that does not mean they are distributing the work in co…

> I think you’re doing away with the fairness exception for criticism?

Perhaps, but it's more an example of the problem: something can be fine at small scale, but cause issues when everyone does it. Tragedy of the commons, but with words.

(From an even more extreme point of view, consider that an image generating AI trained on nothing but photographs taken from drones flying and androids walking all over the place would be able to create photo-realistic images of anything, irregardless of if even one single human artist's works end up in the training set, which in turn means the current concerns about "did the human artists agree to this use" will be quickly made irrelevant because there were none in the training set in the first place).

"Quantity has a quality all its own", whoever really said it first.

> Copyright is intended to reward investment in creative works by giving sole license to distribute. It is not intended to create a monopoly on knowledge about the work.

Sure, but laws change depending on economics. I can easily believe AI will lead to either much stronger or much weaker copyright laws.

Depends who is wielding the power when the change comes.

> If I can ask an LLM (or person!) “what’s the first sentence in Harry Potter?” And then “what’s the second sentence?” and so on, that does not mean they are distributing the work in competition with the rights holders.

Isn't that a description of how BitTorrent works? And The Pirate Bay is kinda infamous for "distributing the work in competition with the rights holders".

> We have gone way overboard with IP protections. The purpose of copyright is served when Rowling buys her 10th mansion. We do not need to further expand copyright to make it illegal to learn from a work or to remember it after reading.

I agree, and was already in favour of radical changes to copyright rules well before LLMs.

(That said, it's more complex because of how hit-driven lots of things are, which means that while nobody needs to defend Rowling's second billion, having looked at the distribution of book sales in the best-seller lists… most of them will/would have need/ed a second source of income to keep publishing).

Re: Training AI models might not need enormous data centres

#56

Post by neom »

I think the problem is people are going to start playing, everyone is going to train in their own things, businesses are going to want to train different architectures for different business functions etc. I did my first real adventure with training last night, $3,200 and a lot of fun later (whooops) - the tooling has become very easy to use, and I presume will just get easier. If I want to train in even say 10ish gigs, wouldn't I want to use a DC, even with a powerful laptop or DiLoCo? Seems unlikely DiLoCo is enough?

(edit: I may also not be accounting enough for using a pre-trained general model next to a fine tuned specialized model?)

Re: Training AI models might not need enormous data centres

#57

Post by aimanbenbaha »

lambda-research wrote: post #50
aimanbenbaha wrote: post #43

This bottleneck right here is why Open Source is presented with a golden plate opportunity to lead the training of cutting edge models. Federated learning breaks the barrier to entry and expands the ecosystem allowing more participants to share compute and/or datasets for small players to train models. DiLoCo introduced by Douillard minimizes communication overhead by averaging weight updates. What this article misse…

> What this article misses though is that despite this, each GPU in the distributed cluster still needs to have enough VRAM to load the entire copy of the model to complete the training process. That's not exactly accurate. In the data parallel side of techniques, the Distributed Data Parallel (DDP) approach does require a fully copy of the model on each GPU. However there's also Fully Sharded Data Parallel (FSDP) wh…

You're right. My caveat not exactly accurate but I wanted to point out where DisTrO might comes in and why it's relevant here.

I mean it reduces the communication overhead by more orders than DiLoCo.

Re: Training AI models might not need enormous data centres

#58

Post by trash_cat »

ghxst wrote: post #24
whazor wrote: post #17

You could consider a LLM as a very lossy compression artifact. Where they took terabytes of input data, and ended up with model under the 100 gigabytes. It is quite remarkable what such a model can do, even fabricating new output that was not in the input data. However, in my naïvety, I wonder whether vastly simpler algorithms could be used to end up with similar results. Regular compression techniques work with spee…

I'm not sure how accurate it is but my gut feeling is that the level of meaningful compression is somehow correlated to the level of intelligence behind a model, I wouldn't be surprised if it ends up being a major focus in general intelligence.

This is the whole premise behind transformers and ChatGPT models and has been discussed by Ilya[0].

[0] https://the-decoder.com/openai-co-founder-explains-the-secre...

Re: Training AI models might not need enormous data centres

#59

Post by lostmsu »

CaptainFever wrote: post #26
Aerroon wrote: post #14

Earlier quoted context omitted.

The internet is for ____. That could be a factor that unites enough people to donate their compute time to build diffusion models. At least if it was easy enough to set up.

Related: people donating computing power to run diffusion and text models, which is definitely largely used for porn. https://stablehorde.net/ Or the large amounts of community efforts (not exactly crowd sourced though) for diffusion fine-tunes and tools! Pony XL, and other uncensored models, for example. I haven't kept up with the rest, because there's just too much.

You don't have to donate, we will pay you for idle time of your gaming GPU: https://borg.games/setup
Post reply on HN