Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will we run out of ML data? Evidence from projecting dataset size trends (2022)

epochai.org

Post reply on HN
Print view View original

21–30 of 126 posts

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#21

Post by replygirl »

visarga wrote: post #18
replygirl wrote: post #5

If we play our cards right, AI could free people up for more valuable pursuits, and the pace of human information production would increase by orders of magnitude

> free people up for more valuable pursuits It won't roll like that. AI will empower people to be more productive but won't free people up because it makes mistakes, can't help itself, and cannot function autonomously. There is no LLM application that is safe for autonomous usage today. How can we go from 0 to 1? I don't see a path. Self driving cars still can't reach L5 to completely remove the need for driver. But…

Yeah, it's not that I think we'll get all the way there, it's a utopia. My expectation is that within 30 years we reduce the work week by a day or two for most people, compensate for our education system's decline, and avoid energy and food crises, and nothing else fundamentally changes

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#22

Post by replygirl »

gumballindie wrote: post #19
replygirl wrote: post #17

Earlier quoted context omitted.

So hopefully we play our cards right, by extracting benefits from AI that overcompensate for the negative impacts like mass unemployment and democratization of intellectual property. If the spoils are distributed in such a way that people's standard of living is maintained or improved, people have more liesure time, which the social sciences have shown will not mean people will just stop working--they'll work less, b…

Unfortunately real life doesn't work the way you describe. People won't be "free" to enjoy more "leisure" time, by "democratizing" the results of their work. Instead, all of this stolen data, will be used to "free" them from jobs and to consolidate corporate control. Say bye to microbusinesses, to freelancers, to indie developers. You know, those people that have been truly free. Similarly, office workers will be "fr…

I agree that not playing our cards right is the default scenario.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#23

Post by visarga »

gumballindie wrote: post #19
replygirl wrote: post #17

Earlier quoted context omitted.

So hopefully we play our cards right, by extracting benefits from AI that overcompensate for the negative impacts like mass unemployment and democratization of intellectual property. If the spoils are distributed in such a way that people's standard of living is maintained or improved, people have more liesure time, which the social sciences have shown will not mean people will just stop working--they'll work less, b…

Unfortunately real life doesn't work the way you describe. People won't be "free" to enjoy more "leisure" time, by "democratizing" the results of their work. Instead, all of this stolen data, will be used to "free" them from jobs and to consolidate corporate control. Say bye to microbusinesses, to freelancers, to indie developers. You know, those people that have been truly free. Similarly, office workers will be "fr…

> All thanks to "democratizing" ip by stealing data.

That horse is already dead. Large models can learn everything, there's nothing that can be done to stop them from learning. It's too easy for them to do it. We can't hold any meaningful IP when models can generate 100 variations only different enough to pass the test. IP is dead. But on its corpse there will grow a new world of applications. We all got new skills, depends on us if we use them or not.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#24

Post by Buttons840 »

sanxiyn wrote: post #20
Buttons840 wrote: post #11

Earlier quoted context omitted.

> AI can generate as much synthetic data as we need, on demand. I don't think this is right. Can I take an untrained LLM (a neural network with random parameters), and have it start generating garbage, and then train the network to produce more of the same and then have it bootstrap itself to intelligence? Of course not. What if I train it just a little bit first? What if I train it until it produces gibberish, but d…

AlphaZero in fact improves based on its own output, but I agree it is a special case and probably not generalizable.

It's RL though. Its output comes, in part, from interaction with an environment. It also has a well defined objective (win games). GTP doesn't have a clear objective other than "do more of this".

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#25

Post by gumballindie »

visarga wrote: post #23
gumballindie wrote: post #19

Earlier quoted context omitted.

Unfortunately real life doesn't work the way you describe. People won't be "free" to enjoy more "leisure" time, by "democratizing" the results of their work. Instead, all of this stolen data, will be used to "free" them from jobs and to consolidate corporate control. Say bye to microbusinesses, to freelancers, to indie developers. You know, those people that have been truly free. Similarly, office workers will be "fr…

> All thanks to "democratizing" ip by stealing data. That horse is already dead. Large models can learn everything, there's nothing that can be done to stop them from learning. It's too easy for them to do it. We can't hold any meaningful IP when models can generate 100 variations only different enough to pass the test. IP is dead. But on its corpse there will grow a new world of applications. We all got new skills,…

Sure. In before people used to say that currency is dead because crypto currency has replaced money already, and already people are using them, and already [insert marketing statement]. I see they now moved on to ai.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#26

Post by gumballindie »

replygirl wrote: post #22
gumballindie wrote: post #19

Earlier quoted context omitted.

Unfortunately real life doesn't work the way you describe. People won't be "free" to enjoy more "leisure" time, by "democratizing" the results of their work. Instead, all of this stolen data, will be used to "free" them from jobs and to consolidate corporate control. Say bye to microbusinesses, to freelancers, to indie developers. You know, those people that have been truly free. Similarly, office workers will be "fr…

I agree that not playing our cards right is the default scenario.

As a technology ai can indeed free people in a productive manner. But it would appear that we started on the wrong footing. Power will be consolidated in the hands of a few at a scale we haven't seen yet. It all depends tho on whether we can regulate how data is collected, at least at the basic level of not infringing copyright.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#27

Post by nologic01 »

Brute force approaches always hit some wall. ML will be no different. In the decades to come it us quite likely that algorithms will develop in directions orthogonal to current approaches. The idea that you improve performance by throwing gazillions of data into gargantuan models might be even come to be seen as laughable.

Keep in mind (pun) that the only real intelligence here is us, and we are pretty good at figuring out when a tool has exhausted its utility.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#28

Post by mkaic »

visarga wrote: post #23
gumballindie wrote: post #19

Earlier quoted context omitted.

Unfortunately real life doesn't work the way you describe. People won't be "free" to enjoy more "leisure" time, by "democratizing" the results of their work. Instead, all of this stolen data, will be used to "free" them from jobs and to consolidate corporate control. Say bye to microbusinesses, to freelancers, to indie developers. You know, those people that have been truly free. Similarly, office workers will be "fr…

> All thanks to "democratizing" ip by stealing data. That horse is already dead. Large models can learn everything, there's nothing that can be done to stop them from learning. It's too easy for them to do it. We can't hold any meaningful IP when models can generate 100 variations only different enough to pass the test. IP is dead. But on its corpse there will grow a new world of applications. We all got new skills,…

I agree! I actually wrote a blogpost about this recently[0], but the TLDR is that ownership is nothing without enforcement, and it has become increasingly difficult to enforce ownership of intellectual property in the modern world — first digital files, then the sharing of those digital files over the Internet, and now generative models that allow people to create high-quality ripoffs of any IP for zero marginal cost. The sheer volume will just be too much to contend with, because you can't sue everyone. In my view, this is a good thing and a long time coming!

[0] https://blog.kaichristensen.com/p/generative-ai-is-the-final...

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#29

Post by haldujai »

visarga wrote: post #16
haldujai wrote: post #13

I wonder if the better question is not how we get more training data but: If we're running out of training data with hallucinations and performance remaining so inadequate (per OpenAI's whitepaper) is an autoregressive transformer the right architecture? Perhaps ongoing work in finetuning will take these models to the next level but ignoring the LLM hype it really does seem like things have plateaued for a while now…

There is still an order of magnitude more organic text. Ilya Sutskever recently said it was still ok. After that, we got to use reinforcement learning (agent GPTs with tools) to generate and self-validate more examples. One "simple" application would be to build a full index of facts in the whole training corpus. Just pass each document to GPT and ask it to extract the facts. Then create an inverted index, with each…

> There is still an order of magnitude more organic text.

Posing this as a thought experiment, agree we still have more data to go. That we are wondering about this suggests that the current approach may be inadequate, i.e. it should not take petabytes of data for a LLM to match the performance of a high school student (for the LLM = AGI folks).

> One "simple" application would be to build a full index of facts in the whole training corpus. Just pass each document to GPT and ask it to extract the facts.

Agree, KG+LLM is a good next step to explore and should address some hallucination issues (see DRAGON from Leskovec and Liang groups). But we're already now talking about architectural changes as I posited.

In any case, where do we get such knowledge graphs (or index of facts)? Some already exist (e.g. Wiki, UMLS) and were created by humans but are clearly inadequate in coverage.

The proposition of using GPT-like models to generate these (i.e. GraphGPT) seems conceptually flawed as GPT does not itself know if a statement is factual or not which is problematic even for humans.

Settled vs controversial is orders of magnitude more complex, how on earth do we do this without human annotation? You can't rely on frequency (i.e. some things were facts for 100 years but all of a sudden they're not anymore and this is not controversial by definition).

The only reason LLMs work as well as they do now is because sheer volume of data (and NTP) makes the noise seem hidden and by definition an autoregressive model should be somewhat impervious to singular factoids (vs a model being grounded by the garbage dump that is CommonCrawl/the internet).

> At least the model won't hallucinate outside the known facts.

Not sure this is a given, even if a model acts as a natural language database of factoids it is probable that it will hallucinate links unless you're strictly grounding output in which case we've just built a colossally over-engineered IR/STS tool.

> One "simple" application

I think what you've posited is actually harder to build than anything that's been achieved thus far with LLMs.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#30

Post by brazzy »

visarga wrote: post #16
haldujai wrote: post #13

I wonder if the better question is not how we get more training data but: If we're running out of training data with hallucinations and performance remaining so inadequate (per OpenAI's whitepaper) is an autoregressive transformer the right architecture? Perhaps ongoing work in finetuning will take these models to the next level but ignoring the LLM hype it really does seem like things have plateaued for a while now…

There is still an order of magnitude more organic text. Ilya Sutskever recently said it was still ok. After that, we got to use reinforcement learning (agent GPTs with tools) to generate and self-validate more examples. One "simple" application would be to build a full index of facts in the whole training corpus. Just pass each document to GPT and ask it to extract the facts. Then create an inverted index, with each…

>After that, we got to use reinforcement learning (agent GPTs with tools) to generate and self-validate more examples.

How would you "self-validate" against hallucinated facts?

What makes self-validation possible are hard external rules that can be evaluated independently and automatically. Like the rules of Chess or Go.

We don't have anything like that for LLMs and what people want to use them for.

Post reply on HN