Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will we run out of ML data? Evidence from projecting dataset size trends (2022)

epochai.org

Post reply on HN
Print view View original

31–40 of 126 posts

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#31

Post by blackbear_ »

Buttons840 wrote: post #11
cs702 wrote: post #2

No. AI can generate as much synthetic data as we need, on demand. Many SOTA models, in fact, are already being trained with synthetic AI-generated data. See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines

> AI can generate as much synthetic data as we need, on demand. I don't think this is right. Can I take an untrained LLM (a neural network with random parameters), and have it start generating garbage, and then train the network to produce more of the same and then have it bootstrap itself to intelligence? Of course not. What if I train it just a little bit first? What if I train it until it produces gibberish, but d…

> Because so far we've never seen an AI improve based on its own output.

Maybe it's because AI is such an overloaded term, but this is pretty commonplace for (semi-)supervised learning algorithms.

Pseudo-labeling [1,2] is an example of this that has been around for decades. When done properly it does improve the performance of the original model, up to a certain limit (far from the singularity).

Moreover, it is apparently possible to improve a model's performance by augmenting it's training set with synthetic examples generated by a second model [3].

Finally, boosting [4] can also be seen as iteratively leveraging the output of a model to train a slightly better model. In fact, a specific type of boosting often yields state of the art performance on tabular data.

[1] https://arxiv.org/abs/2101.06329

[2] https://stats.stackexchange.com/questions/364584/why-does-us...

[3] https://arxiv.org/abs/2304.08466

[4] https://en.m.wikipedia.org/wiki/Boosting_(machine_learning)

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#32

Post by hackerlight »

cs702 wrote: post #2

No. AI can generate as much synthetic data as we need, on demand. Many SOTA models, in fact, are already being trained with synthetic AI-generated data. See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines

> AI can generate as much synthetic data as we need, on demand.

Doesn't work in majority of domains. You need to know the generating process (e.g. game rules) and build a realistic simulation environment that emulates that, in order to generate data that is useful. Both of these things are out of reach for most applications.

I believe the next large step will be multi-modal, where text is contextualized by video so the LLM will be able to concretize what "sitting on a chair" actually means with a single example, without needing to see thousands of textual associations to infer the meaning from the text.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#33

Post by haldujai »

blackbear_ wrote: post #31
Buttons840 wrote: post #11

Earlier quoted context omitted.

> AI can generate as much synthetic data as we need, on demand. I don't think this is right. Can I take an untrained LLM (a neural network with random parameters), and have it start generating garbage, and then train the network to produce more of the same and then have it bootstrap itself to intelligence? Of course not. What if I train it just a little bit first? What if I train it until it produces gibberish, but d…

> Because so far we've never seen an AI improve based on its own output. Maybe it's because AI is such an overloaded term, but this is pretty commonplace for (semi-)supervised learning algorithms. Pseudo-labeling [1,2] is an example of this that has been around for decades. When done properly it does improve the performance of the original model, up to a certain limit (far from the singularity). Moreover, it is appar…

This really only works well in resource limited settings and/or semisupervised tasks.

I've tried augmentation for LLM domain adaptation and it's very modest gains in the best of situations, and even still the augmented corpus is a very tiny fraction of the underlying training corpus.

I believe OP's question was getting at whether synthetic data is useful as a substantial corpus for unsupervised training of a language model (given the topic it's reasonable to disregard other areas of 'AI') and that answer appears to be no or at least unproven and non-intuitive.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#34

Post by blibble »

replygirl wrote: post #5

If we play our cards right, AI could free people up for more valuable pursuits, and the pace of human information production would increase by orders of magnitude

> the pace of human information production would increase by orders of magnitude

you mean boilerplate and spam right?

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#35

Post by jacooper »

replygirl wrote: post #21
visarga wrote: post #18

Earlier quoted context omitted.

> free people up for more valuable pursuits It won't roll like that. AI will empower people to be more productive but won't free people up because it makes mistakes, can't help itself, and cannot function autonomously. There is no LLM application that is safe for autonomous usage today. How can we go from 0 to 1? I don't see a path. Self driving cars still can't reach L5 to completely remove the need for driver. But…

Yeah, it's not that I think we'll get all the way there, it's a utopia. My expectation is that within 30 years we reduce the work week by a day or two for most people, compensate for our education system's decline, and avoid energy and food crises, and nothing else fundamentally changes

Assuming this happens without any violence, which I truly doubt. Huge socioeconomic changes like this always come as a result of violence and uproar.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#36

Post by ChatGTP »

replygirl wrote: post #21
visarga wrote: post #18

Earlier quoted context omitted.

> free people up for more valuable pursuits It won't roll like that. AI will empower people to be more productive but won't free people up because it makes mistakes, can't help itself, and cannot function autonomously. There is no LLM application that is safe for autonomous usage today. How can we go from 0 to 1? I don't see a path. Self driving cars still can't reach L5 to completely remove the need for driver. But…

Yeah, it's not that I think we'll get all the way there, it's a utopia. My expectation is that within 30 years we reduce the work week by a day or two for most people, compensate for our education system's decline, and avoid energy and food crises, and nothing else fundamentally changes

I can’t see us going from Microsoft and Open AI stealing everyone’s work and selling it without attribution or respect for GPL (for example ) to technological utopia anytime soon.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#37

Post by gleenn »

nologic01 wrote: post #27

Brute force approaches always hit some wall. ML will be no different. In the decades to come it us quite likely that algorithms will develop in directions orthogonal to current approaches. The idea that you improve performance by throwing gazillions of data into gargantuan models might be even come to be seen as laughable. Keep in mind (pun) that the only real intelligence here is us, and we are pretty good at figuri…

AI had a winter of many decades because the hardware wasn't there and there were better alternatives, especially for neural nets. Now ChatGPT etc comes out, with unbelievable results, decades in the making. And a couple months we're already writing it off because of the next limitation? Maybe let's give it more than a month or two to figure out if we even need all that data. I heard they're already talking about trying to significantly reduce the model hyper parameters size even though a large model size increase apparently the reason GPT 4 was so much better than 3. Give it a minute IMHO before making generalizations like this so soon

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#38

Post by pizza »

brazzy wrote: post #30
visarga wrote: post #16

Earlier quoted context omitted.

There is still an order of magnitude more organic text. Ilya Sutskever recently said it was still ok. After that, we got to use reinforcement learning (agent GPTs with tools) to generate and self-validate more examples. One "simple" application would be to build a full index of facts in the whole training corpus. Just pass each document to GPT and ask it to extract the facts. Then create an inverted index, with each…

>After that, we got to use reinforcement learning (agent GPTs with tools) to generate and self-validate more examples. How would you "self-validate" against hallucinated facts? What makes self-validation possible are hard external rules that can be evaluated independently and automatically. Like the rules of Chess or Go. We don't have anything like that for LLMs and what people want to use them for.

RLHF seems to suggest that human feedback to tune the model after plain textual data pretraining is quite potent per sample. There might be some optimal ratio of data+model size:rlhf size that works quite favorably for us in getting hallucinations to a minimum. Furthermore there might be some “there” there, in the hallucinations, that has yet to be identified as valuable in itself. Either way it seems like our ability to wrangle these models is getting better

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#39

Post by zone411 »

Buttons840 wrote: post #11
cs702 wrote: post #2

No. AI can generate as much synthetic data as we need, on demand. Many SOTA models, in fact, are already being trained with synthetic AI-generated data. See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines

> AI can generate as much synthetic data as we need, on demand. I don't think this is right. Can I take an untrained LLM (a neural network with random parameters), and have it start generating garbage, and then train the network to produce more of the same and then have it bootstrap itself to intelligence? Of course not. What if I train it just a little bit first? What if I train it until it produces gibberish, but d…

My generative melody models have done it for over a year but that's with human curation, so it's not self-improving. It's typically easier to curate than to generate, and it's especially true for music and images. It's much simpler to recognize a good melody than to compose a new one. The same applies to writing, but to a lesser extent.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#40

Post by airgapstopgap »

nologic01 wrote: post #27

Brute force approaches always hit some wall. ML will be no different. In the decades to come it us quite likely that algorithms will develop in directions orthogonal to current approaches. The idea that you improve performance by throwing gazillions of data into gargantuan models might be even come to be seen as laughable. Keep in mind (pun) that the only real intelligence here is us, and we are pretty good at figuri…

We won't hit the wall.

Somewhat counterintuitively, scaling datasets is the lazy and economical approach. If you have the compute already, might as well dig an OOM more text tokens.

But there are other sources of data, and slightly different ways to utilize it. Multimodality, in very large training runs, will almost inevitably increase sample efficiency (for obvious reasons of context richness), synthetic data is already very effective [1], and there are and will be discovered other ways to do more in the condition of diminishing raw text resources. But a thorough abandonment of the scaling strategy is very unlikely.

Sutton's Bitter Lesson [2] points at a very powerful rule of thumb: we shouldn't turn AI engineering into a contest of smartness, we should allow complex smartness to emerge from generic low-level algorithms. What will be seen as laughable in decades to come is not the scaling strategy, but the Godlike conceit of people who thought they can devise generally applicable rules of reasoning from first principles.

1: https://arxiv.org/abs/2304.08466 2: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Post reply on HN