Will we run out of ML data? Evidence from projecting dataset size trends (2022)
1–10 of 126 posts
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#2No.
AI can generate as much synthetic data as we need, on demand.
Many SOTA models, in fact, are already being trained with synthetic AI-generated data.
See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#3No. AI can generate as much synthetic data as we need, on demand. Many SOTA models, in fact, are already being trained with synthetic AI-generated data. See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines
Lots of reasons this isn't universally true - it only works if you know enough about the data to simulate it, and your stuck within some distribution + human guesses space that's not all encompassing.
The easiest counterexample is training LLMs, how are you going to synthesize useful language examples if you want more. Some version of this is true for most applications.
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#4No. AI can generate as much synthetic data as we need, on demand. Many SOTA models, in fact, are already being trained with synthetic AI-generated data. See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines
You're just sampling from an already sampled distribution.
This is not the same thing. There will still be value for fine tuning, but it's no substitute.
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#5If we play our cards right, AI could free people up for more valuable pursuits, and the pace of human information production would increase by orders of magnitude
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#6No. AI can generate as much synthetic data as we need, on demand. Many SOTA models, in fact, are already being trained with synthetic AI-generated data. See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines
You're just sampling from an already sampled distribution. This is not the same thing. There will still be value for fine tuning, but it's no substitute.
It's not one way consumer just like humans are not. It can direct long term evolution of reason. For starters it can be used to denoise/dedup/optimise training set to be closer to optimum (to create smaller "copies" of itself).
There are instances of things that happened (history, what Paris Hilton did say on 22nd of April etc, big database of mostly irrelevant facts) and truths (math, physics, chemistry etc) where AI can enhance discoveries by helping us to see what we have not yet realised.
Both seem endless tbh but personally I'm more interested in latter.
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#7This analysis misses the impact of AI models being deployed, like is happening rapidly right now. Production applications built on AI will provide ample (infinite?) additional training data to feed back into the underlying models.
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#8No. AI can generate as much synthetic data as we need, on demand. Many SOTA models, in fact, are already being trained with synthetic AI-generated data. See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines
Lots of reasons this isn't universally true - it only works if you know enough about the data to simulate it, and your stuck within some distribution + human guesses space that's not all encompassing. The easiest counterexample is training LLMs, how are you going to synthesize useful language examples if you want more. Some version of this is true for most applications.
Yeah the issue is you can generate data, but it won’t be good data. Training over random strings won’t make you learn language, but it’s technically data.
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#9This analysis misses the impact of AI models being deployed, like is happening rapidly right now. Production applications built on AI will provide ample (infinite?) additional training data to feed back into the underlying models.
Not sure that synthetic or LLM-generated training data is as useful as human generated text.
It seems "good enough" (for now) but synthetic makes up a very small proportion of the training set being used in current models that have been trained on it, if that proportion ends up being mostly synthetic we'll likely see whatever weird hallucinations and biases in the dominant backend (GPT4 or whatever) become amplified.
It's been shown repeatedly that garbage in = garbage out for training data.
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#10No. AI can generate as much synthetic data as we need, on demand. Many SOTA models, in fact, are already being trained with synthetic AI-generated data. See https://en.wikipedia.org/wiki/Betteridge's_law_of_headlines
To my knowledge no SOTA model has been trained on a significant proportion of synthetic data, has this changed?
The best examples I know of are instruction tuning sets but that is a minute amount of data compared to the unsupervised training data.