Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will we run out of ML data? Evidence from projecting dataset size trends (2022)

epochai.org

Post reply on HN
Print view View original

11–20 of 126 posts

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#11

Post by Buttons840 »

cs702 wrote: post #2

No. AI can generate as much synthetic data as we need, on demand. Many SOTA models, in fact, are already being trained with synthetic AI-generated data. See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines

> AI can generate as much synthetic data as we need, on demand.

I don't think this is right.

Can I take an untrained LLM (a neural network with random parameters), and have it start generating garbage, and then train the network to produce more of the same and then have it bootstrap itself to intelligence? Of course not.

What if I train it just a little bit first? What if I train it until it produces gibberish, but does occasionally string two words together that are spelled correctly. Can I have it produce petabytes of gibberish and then train on that to reach GTP4's level?

You seem to argue that at some point, the AI is able to improve by training on its own output. At what point does that arrive? Because so far we've never seen an AI improve based on its own output. (As far as I know?)

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#12

Post by gumballindie »

replygirl wrote: post #5

If we play our cards right, AI could free people up for more valuable pursuits, and the pace of human information production would increase by orders of magnitude

> free people

As opposed to what? Being "captive" in jobs for paying bills?

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#13

Post by haldujai »

I wonder if the better question is not how we get more training data but:

If we're running out of training data with hallucinations and performance remaining so inadequate (per OpenAI's whitepaper) is an autoregressive transformer the right architecture?

Perhaps ongoing work in finetuning will take these models to the next level but ignoring the LLM hype it really does seem like things have plateaued for a while now (with expected gains from scaling).

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#14

Post by digdugdirk »

gumballindie wrote: post #12
replygirl wrote: post #5

If we play our cards right, AI could free people up for more valuable pursuits, and the pace of human information production would increase by orders of magnitude

> free people As opposed to what? Being "captive" in jobs for paying bills?

I mean... Yes.

What would you suggest as the alternative?

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#15

Post by gumballindie »

digdugdirk wrote: post #14
gumballindie wrote: post #12

Earlier quoted context omitted.

> free people As opposed to what? Being "captive" in jobs for paying bills?

I mean... Yes. What would you suggest as the alternative?

Erm not causing mass unemployment by stealing data? Also people go freely where there's pay. Seems like there aren't many opportunities and there will fewer.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#16

Post by visarga »

haldujai wrote: post #13

I wonder if the better question is not how we get more training data but: If we're running out of training data with hallucinations and performance remaining so inadequate (per OpenAI's whitepaper) is an autoregressive transformer the right architecture? Perhaps ongoing work in finetuning will take these models to the next level but ignoring the LLM hype it really does seem like things have plateaued for a while now…

There is still an order of magnitude more organic text. Ilya Sutskever recently said it was still ok. After that, we got to use reinforcement learning (agent GPTs with tools) to generate and self-validate more examples.

One "simple" application would be to build a full index of facts in the whole training corpus. Just pass each document to GPT and ask it to extract the facts. Then create an inverted index, with each fact and its references. This will allow us to generate a wikipedia-like corpus of exhaustive fact research. We can say if a fact is known or not, we can tell if it is settled or controversial, and if it is a preference we can tell what is the distribution. This has got to help with factuality and generate lots of text to feed the model. Basically only costs electricity and GPU. It nicely side-steps the problem of truth by simply modelling the empirical distribution in an explicit way. At least the model won't hallucinate outside the known facts.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#17

Post by replygirl »

gumballindie wrote: post #15
digdugdirk wrote: post #14

Earlier quoted context omitted.

I mean... Yes. What would you suggest as the alternative?

Erm not causing mass unemployment by stealing data? Also people go freely where there's pay. Seems like there aren't many opportunities and there will fewer.

So hopefully we play our cards right, by extracting benefits from AI that overcompensate for the negative impacts like mass unemployment and democratization of intellectual property. If the spoils are distributed in such a way that people's standard of living is maintained or improved, people have more liesure time, which the social sciences have shown will not mean people will just stop working--they'll work less, but with higher productivity on things promising a greater benefit to family, community, and society.

Forgive me if I'm misreading, but I'm having trouble with your line of reasoning. Your first reply to me scarequoting "captive" strongly implies an argument that the imperative to seek employment for survival is not a limiting factor on how people spend their time, and therefore that my suggestion that giving people more choice over how they apply their talents could be a good thing is irrelevant; but your child reply implies a concern that AI taking over some human labor will cause mass unemployment and explicitly states choice is declining.

I'm advocating that, since the genie is out of the bottle, AI could be used to free people from toil, just as other labor innovations like machinery and the 40-hour work week have done. Why the dismissive snark? In the abstract, do we not want the same thing?

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#18

Post by visarga »

replygirl wrote: post #5

If we play our cards right, AI could free people up for more valuable pursuits, and the pace of human information production would increase by orders of magnitude

> free people up for more valuable pursuits

It won't roll like that. AI will empower people to be more productive but won't free people up because it makes mistakes, can't help itself, and cannot function autonomously. There is no LLM application that is safe for autonomous usage today. How can we go from 0 to 1? I don't see a path. Self driving cars still can't reach L5 to completely remove the need for driver.

But maybe this is a blessing in disguise. It will make AI more like a new ability of humans than of the companies. Companies need people to unlock AI efficiencies. And AI tends to become open sourced so everyone has access to the same. AI is not a moat for companies and human ability to hand-held it is tied to individuals. That would make the transition easier. Solving that last 1% accuracy might encounter exponential friction and last for a while.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#19

Post by gumballindie »

replygirl wrote: post #17
gumballindie wrote: post #15

Earlier quoted context omitted.

Erm not causing mass unemployment by stealing data? Also people go freely where there's pay. Seems like there aren't many opportunities and there will fewer.

So hopefully we play our cards right, by extracting benefits from AI that overcompensate for the negative impacts like mass unemployment and democratization of intellectual property. If the spoils are distributed in such a way that people's standard of living is maintained or improved, people have more liesure time, which the social sciences have shown will not mean people will just stop working--they'll work less, b…

Unfortunately real life doesn't work the way you describe. People won't be "free" to enjoy more "leisure" time, by "democratizing" the results of their work. Instead, all of this stolen data, will be used to "free" them from jobs and to consolidate corporate control. Say bye to microbusinesses, to freelancers, to indie developers. You know, those people that have been truly free. Similarly, office workers will be "free" to lose the jobs that they chose to perform, and will have nowhere else to go but unemployment lines. All thanks to "democratizing" ip by stealing data.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#20

Post by sanxiyn »

Buttons840 wrote: post #11
cs702 wrote: post #2

No. AI can generate as much synthetic data as we need, on demand. Many SOTA models, in fact, are already being trained with synthetic AI-generated data. See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines

> AI can generate as much synthetic data as we need, on demand. I don't think this is right. Can I take an untrained LLM (a neural network with random parameters), and have it start generating garbage, and then train the network to produce more of the same and then have it bootstrap itself to intelligence? Of course not. What if I train it just a little bit first? What if I train it until it produces gibberish, but d…

AlphaZero in fact improves based on its own output, but I agree it is a special case and probably not generalizable.
Post reply on HN