Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Will we run out of ML data? Evidence from projecting dataset size trends (2022)

epochai.org

Post reply on HN
Print view View original

41–50 of 126 posts

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#41

Post by zarzavat »

visarga wrote: post #16
haldujai wrote: post #13

I wonder if the better question is not how we get more training data but: If we're running out of training data with hallucinations and performance remaining so inadequate (per OpenAI's whitepaper) is an autoregressive transformer the right architecture? Perhaps ongoing work in finetuning will take these models to the next level but ignoring the LLM hype it really does seem like things have plateaued for a while now…

There is still an order of magnitude more organic text. Ilya Sutskever recently said it was still ok. After that, we got to use reinforcement learning (agent GPTs with tools) to generate and self-validate more examples. One "simple" application would be to build a full index of facts in the whole training corpus. Just pass each document to GPT and ask it to extract the facts. Then create an inverted index, with each…

After the low hanging fruit - the high quality data such as scientific papers, libgen, stackexchange, wikipedia, etc — has been exhausted, that’s it. There’s no more data of that kind. There’s not 9 other wikipedias or 9 other libgens. There is only a certain quantity of high-quality codified knowledge in existence and models need to be able to deal with that constraint. Feeding it more and more lower quality text is not going to improve performance because we already fed it all the text that we use. There’s a reason that PhDs don’t involve reading tumblr all day.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#43

Post by majikaja »

flyval wrote: post #42

This is dumb. An individual human takes in more data than modern LLMs do. https://open.substack.com/pub/echoesofid/p/why-llms-struggle...

>Just the vision data of a baby’s first year easily adds up to petabytes

What encoding is this??

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#44

Post by lostmsu »

majikaja wrote: post #43
flyval wrote: post #42

This is dumb. An individual human takes in more data than modern LLMs do. https://open.substack.com/pub/echoesofid/p/why-llms-struggle...

>Just the vision data of a baby’s first year easily adds up to petabytes What encoding is this??

Uncompressed 2x 8k by 8k 24bpp 24FPS video. Comes at about 500GB per hour.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#45

Post by sanxiyn »

airgapstopgap wrote: post #40
nologic01 wrote: post #27

Brute force approaches always hit some wall. ML will be no different. In the decades to come it us quite likely that algorithms will develop in directions orthogonal to current approaches. The idea that you improve performance by throwing gazillions of data into gargantuan models might be even come to be seen as laughable. Keep in mind (pun) that the only real intelligence here is us, and we are pretty good at figuri…

We won't hit the wall. Somewhat counterintuitively, scaling datasets is the lazy and economical approach. If you have the compute already, might as well dig an OOM more text tokens. But there are other sources of data, and slightly different ways to utilize it. Multimodality, in very large training runs, will almost inevitably increase sample efficiency (for obvious reasons of context richness), synthetic data is alr…

I don't think your "synthetic data on ImageNet" reference shows "synthetic data is already very effective". Since many people won't read the paper, here's what it says:

Training ResNet-50 on real ImageNet gives 73.09% top-1 accuracy, while training it on synthetic data (same resolution, same number of images) generated by this work gives 64.96%, which is SOTA compared to previous work's 63.02%. Therefore, synthetic data is worse than real data for now.

But synthetic data is not useless, because training on real data plus synthetic data is a bit better than both real data and synthetic data. (Accuracy here is different due to different methodology.) Using 1:1 real data and synthetic data improves accuracy from 76.39% to 77.61%. But using 1:2 is worse than 1:1 (77.16%), even if dataset became 50% larger. With 1:4, result is worse than not using synthetic data at all. So synthetic data at best can enlarge dataset by 5x, more likely just 2x.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#46

Post by majikaja »

lostmsu wrote: post #44
majikaja wrote: post #43

Earlier quoted context omitted.

>Just the vision data of a baby’s first year easily adds up to petabytes What encoding is this??

Uncompressed 2x 8k by 8k 24bpp 24FPS video. Comes at about 500GB per hour.

Maybe if we store text data as sequences of 10k x 10k PNGs (one for each letter) and add an image recognition layer it would improve LLM perf

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#47

Post by amscanne »

lostmsu wrote: post #44
majikaja wrote: post #43

Earlier quoted context omitted.

>Just the vision data of a baby’s first year easily adds up to petabytes What encoding is this??

Uncompressed 2x 8k by 8k 24bpp 24FPS video. Comes at about 500GB per hour.

It’s always interesting to think about the exact technological analog for our biological sensors, but I believe that our vision would be way less than that in terms of raw data. We have a super high-res area at the center of vision (the fovea), but the rest is extremely low resolution (but with high movement and light sensitivity).

I think we could reasonably say that if an optical nerve has 1mm neurons on average, and they can fire at 250Hz at the most, that’s 250mbps or ~31mb/s per eye of uncompressed data as an upper bound.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#49

Post by StrangeATractor »

On this note, the data-set available if you start collecting today is tainted with experimental AI content. Not the biggest issue right now but as time goes on this problem will get worse and we'll be basing our simulations of intelligence on the output of our simulations of intelligence, a brave new abstraction.

Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)

#50

Post by nologic01 »

airgapstopgap wrote: post #40
nologic01 wrote: post #27

Brute force approaches always hit some wall. ML will be no different. In the decades to come it us quite likely that algorithms will develop in directions orthogonal to current approaches. The idea that you improve performance by throwing gazillions of data into gargantuan models might be even come to be seen as laughable. Keep in mind (pun) that the only real intelligence here is us, and we are pretty good at figuri…

We won't hit the wall. Somewhat counterintuitively, scaling datasets is the lazy and economical approach. If you have the compute already, might as well dig an OOM more text tokens. But there are other sources of data, and slightly different ways to utilize it. Multimodality, in very large training runs, will almost inevitably increase sample efficiency (for obvious reasons of context richness), synthetic data is alr…

You are masquerading personal preferences (and possibly professional interests) as rules of nature. If anything, Godlike conceit definetely applies to some ML accolytes.

In any case, with your last point "we should allow complex smartness to emerge" you essentially agree with my point that new levels will emerge from orthogonal (new) directions.

The good thing about brute force is that it summons so many resources it primes the way for smarter approaches.

For those not conceited the objective is not some deus-ex-machina but "algorithms that work".

Post reply on HN