Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Training AI models might not need enormous data centres

economist.com

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Training AI models might not need enormous data centres

#31

Post by purplethinking »

godelski wrote: post #23
whazor wrote: post #17

You could consider a LLM as a very lossy compression artifact. Where they took terabytes of input data, and ended up with model under the 100 gigabytes. It is quite remarkable what such a model can do, even fabricating new output that was not in the input data. However, in my naïvety, I wonder whether vastly simpler algorithms could be used to end up with similar results. Regular compression techniques work with spee…

> However, in my naïvety, I wonder whether vastly simpler algorithms could be used to end up with similar results. Almost certainly. Distillation demonstrates this. The difficulty is training. It's harder to train a smaller network and harder to train with less data. But look at humans, they ingest far less data and certainly less diverse data. We are extremely computationally efficient. I guess you have to be when y…

> they ingest far less data

True in terms of text, but not if you include video, audio, touch etc. Sure, one could argue that there is much less information content in video than their raw bytes, but even so, we spend many years building a world model as we play with tools, exist in the world and go to school. I don't deny humans are more efficient learners but people tend to forget this. Also, children are taught things in ascending order of difficulty, while with LLMs we just throw random pieces of text at it. There is sure to be a lot of progress in curriculum learning for AI models.

Re: Training AI models might not need enormous data centres

#32

Post by ben_w »

dbspin wrote: post #22
myrmidon wrote: post #21

Earlier quoted context omitted.

I think it's important to remember that we know neural networks can be trained to a very useful state from scratch for 24 GJ: This is 25 W for 30 years (or 7000 kWh, or a good half ton of diesel fuel), which is what a human brain consumes until adulthood. Even though our artificial training efficiency is worse now, likely to stay worse because we want to trade efficiency for faster training, and because we want to cr…

This misses that evolution has been pre-training the human cognitive architecture - brain, limbic system, sympathetic and parasympathetic nervous systems, coevolved viral and bacterial ecosystems - for millions of years. We're not a tabula rasa training at birth to perfectly fit whatever set of training data we're presented. Far from it. Human learning is more akin to RAG, or test time training - specialising a heavi…

> This misses that evolution has been pre-training the human cognitive architecture - brain, limbic system, sympathetic and parasympathetic nervous systems, coevolved viral and bacterial ecosystems - for millions of years. We're not a tabula rasa training at birth to perfectly fit whatever set of training data we're presented. Far from it. Human learning is more akin to RAG

Yes, but.

The human genome isn't that big (3.1 gigabases), and most of that is shared with other species that aren't anything like as intelligent — it's full of stuff that keeps us physically alive, lets us digest milk as adults, darkens our skin when exposed to too much UV so we don't get cancer, gives us (usually) four limbs with (usually) five digits that have keratin plates on their tips, etc.

That pre-training likely gives us innate knowledge of smiles and laughter, of the value judgment that pain is bad and that friendship is good, and (I suspect from my armchair) enough* of a concept of gender that when we hit puberty we're not all bisexual by default.

Also, there's nothing stopping someone from donating their genome to be used as a pre-training system, if we could decode the genome well enough to map out pre-training like that.

* which may be some proxy for it, e.g. "arousal = ((smell exogenous sex hormone) and (exogenous hormone xor endogenous hormone))", which then gets used to train the rest of our brains for specific interests — evolution is full of hack jobs like that

Re: Training AI models might not need enormous data centres

#33

Post by rcxdude »

ghxst wrote: post #25
HPsquared wrote: post #15

Earlier quoted context omitted.

Unfortunately, LLM training is not as computationally easy (embarrassingly parallel) as mining bitcoins.

If that were to be solved (if at all possible, and feasible / competitive) I can definitely see "LLM mining" be a historic milestone. Also much closer to the spirit of F@H in some sense, depending how you look at it. Would there be a financial incentive? And how would it be distributed? Could you receive a stake in the LLM proportional to the contribution you did? Would that be similar in some sense to purchasing sto…

This would require a revolution in the algorithms used to train a neural net: currently LLM training is at best distributed amongst GPUs in racks in the same datacenter, and ideally nearby racks, and that's already a significant engineering challenge, because each step needs to work from the step before, and each step updates all of the weights, so it's hard to parallelise. You can do it a little bit, because you can e.g. do a little bit of training with part of the dataset on one part of the cluster, and another part elsewhere, but this doesn't scale linearly (i.e. you need more compute overall to get the model to converge to something useful), and you still need a lot of bandwidth between your nodes to synchronize the networks frequently.

All of this makes it very poorly suited to a collection of heterogeneous compute connected via the internet, which wants a large chunk of mostly independent tasks which have a high compute cost but relatively low bandwidth requirements.

Re: Training AI models might not need enormous data centres

#34

Post by dbspin »

ben_w wrote: post #32
dbspin wrote: post #22

Earlier quoted context omitted.

This misses that evolution has been pre-training the human cognitive architecture - brain, limbic system, sympathetic and parasympathetic nervous systems, coevolved viral and bacterial ecosystems - for millions of years. We're not a tabula rasa training at birth to perfectly fit whatever set of training data we're presented. Far from it. Human learning is more akin to RAG, or test time training - specialising a heavi…

> This misses that evolution has been pre-training the human cognitive architecture - brain, limbic system, sympathetic and parasympathetic nervous systems, coevolved viral and bacterial ecosystems - for millions of years. We're not a tabula rasa training at birth to perfectly fit whatever set of training data we're presented. Far from it. Human learning is more akin to RAG Yes, but. The human genome isn't that big (…

You've missed the fact that sequencing our genome isn't gathering all the information required. To duplicate a human in computational space - say to create some accelerated AI simulation, you'd need to sequence a complete Telomere-to-Telomere genome (something achieved for the first time only last year!), complete Centromere sequencing (not yet achieved). You'd also need to 'sequence' or somehow encode the epigenome - DNA methylation, histone modifications, and other epigenetic markers. Then you'd need to do the same for both mitochondrial DNA and the human microbiome - every functional bacteria and virus we host (quite the task given how little we understand this ecosystem and its interactions with our own behaviour). Then you'd need to combine genome sequencing with transcriptomics (RNA sequencing), proteomics (proteins), and metabolomics to get a holistic view of human biology.

To make this data 'actionable' for a synthetic intelligence you'd need to functionally replicate the contributions of the intrauterine environment to development, and lastly simulate the social and physical environment. This can't be 'decoded' in the way you implicitly suggest - since it's decompression is computationally irreducible. These are dynamic processes that need to be undergone in order to create the fully developed individual.

[1] https://www.bbc.com/future/article/20230210-the-man-whose-ge...

Re: Training AI models might not need enormous data centres

#35

Post by red75prime »

whazor wrote: post #17

You could consider a LLM as a very lossy compression artifact. Where they took terabytes of input data, and ended up with model under the 100 gigabytes. It is quite remarkable what such a model can do, even fabricating new output that was not in the input data. However, in my naïvety, I wonder whether vastly simpler algorithms could be used to end up with similar results. Regular compression techniques work with spee…

The remarkable thing about this compression method is that stochastic gradient descent for some reason creates algorithms in the network. Not Turing-complete algorithms, of course, but algorithms nevertheless.

An LLM trained on the addition and multiplication data develops circuits for addition and multiplication[1].

It stands to reason that LLM trained on human-produced data develop algorithms that try to approximate the data production process (within their computational limits).

[1] https://arxiv.org/abs/2308.01154

Re: Training AI models might not need enormous data centres

#36

Post by EGreg »

dbspin wrote: post #22
myrmidon wrote: post #21

Earlier quoted context omitted.

I think it's important to remember that we know neural networks can be trained to a very useful state from scratch for 24 GJ: This is 25 W for 30 years (or 7000 kWh, or a good half ton of diesel fuel), which is what a human brain consumes until adulthood. Even though our artificial training efficiency is worse now, likely to stay worse because we want to trade efficiency for faster training, and because we want to cr…

This misses that evolution has been pre-training the human cognitive architecture - brain, limbic system, sympathetic and parasympathetic nervous systems, coevolved viral and bacterial ecosystems - for millions of years. We're not a tabula rasa training at birth to perfectly fit whatever set of training data we're presented. Far from it. Human learning is more akin to RAG, or test time training - specialising a heavi…

It’s like I’m talking to Chomsky again … :)

Re: Training AI models might not need enormous data centres

#37

Post by mr_toad »

HPsquared wrote: post #27
ghxst wrote: post #25

Earlier quoted context omitted.

If that were to be solved (if at all possible, and feasible / competitive) I can definitely see "LLM mining" be a historic milestone. Also much closer to the spirit of F@H in some sense, depending how you look at it. Would there be a financial incentive? And how would it be distributed? Could you receive a stake in the LLM proportional to the contribution you did? Would that be similar in some sense to purchasing sto…

The models are too large to fit on a desktop GPU's VRAM. Progress would either require smaller models (MoE might help here? not sure) or bigger VRAM. For example training a 70 billion parameter model would require at least 140GB of VRAM in each system, whereas a large desktop GPU (4090) has only 24GB. You need enough memory to run the unquantized model for training, then stream the training data through - that part i…

Data parallel training is not the only approach. Sometimes the model itself needs to be distributed across multiple GPU.

https://www.microsoft.com/en-us/research/blog/zero-deepspeed...

The communications overhead of doing this over the internet might be unworkable though.

Re: Training AI models might not need enormous data centres

#38

Post by agentultra »

TZubiri wrote: post #29
grumbelbart2 wrote: post #7

Earlier quoted context omitted.

They could, would and should. But: Training a state of the art LLM costs millions in GPU, electricity alone. There is no "open" organization at this point that can cover this. Current "open source public models" are shared by big players like Meta to undermine the competition. And they only publish their weights, not the training data, training protocols, training code; meaning it's not reproducible, and questionable…

Asking to share the training data is a bit too much, it's petabytes of data, probably has privacy implications. You can study and reproduce with your own training data right?

Probably legals ones too. Such aa evidence of copyright infringement.

Re: Training AI models might not need enormous data centres

#39

Post by htrp »

mr_toad wrote: post #37
HPsquared wrote: post #27

Earlier quoted context omitted.

The models are too large to fit on a desktop GPU's VRAM. Progress would either require smaller models (MoE might help here? not sure) or bigger VRAM. For example training a 70 billion parameter model would require at least 140GB of VRAM in each system, whereas a large desktop GPU (4090) has only 24GB. You need enough memory to run the unquantized model for training, then stream the training data through - that part i…

Data parallel training is not the only approach. Sometimes the model itself needs to be distributed across multiple GPU. https://www.microsoft.com/en-us/research/blog/zero-deepspeed... The communications overhead of doing this over the internet might be unworkable though.

or if the internet became significantly faster fiber connections

Re: Training AI models might not need enormous data centres

#40

Post by htrp »

sebmellen wrote: post #8
grumbelbart2 wrote: post #7

Earlier quoted context omitted.

They could, would and should. But: Training a state of the art LLM costs millions in GPU, electricity alone. There is no "open" organization at this point that can cover this. Current "open source public models" are shared by big players like Meta to undermine the competition. And they only publish their weights, not the training data, training protocols, training code; meaning it's not reproducible, and questionable…

Doesn’t Deepseek somewhat counter this narrative?

Don't they have something like 10k plus current gen GPUs?
Post reply on HN