Earlier quoted context omitted.
Blind kids don't though, and they still end up being smarter than GPT-4
They still take in a lot of sensory data, eg related to touch and proprioception.
True but surely that doesn't come to terabytes of data
Will we run out of ML data? Evidence from projecting dataset size trends (2022)
91–100 of 126 posts
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#92Earlier quoted context omitted.
Uncompressed 2x 8k by 8k 24bpp 24FPS video. Comes at about 500GB per hour.
It’s always interesting to think about the exact technological analog for our biological sensors, but I believe that our vision would be way less than that in terms of raw data. We have a super high-res area at the center of vision (the fovea), but the rest is extremely low resolution (but with high movement and light sensitivity). I think we could reasonably say that if an optical nerve has 1mm neurons on average, a…
I don't think you correctly calculate bandwidth in this case. You assume 1 bit per neuron per tick, but time when it fires within the tick also matters, and that information is missing from multipliers.
Also, there's no reason to use data from optical nerves as input, as it is already precompressed. You should be counting optical receptors instead (120 000 000).
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#93Earlier quoted context omitted.
Yeah, it's not that I think we'll get all the way there, it's a utopia. My expectation is that within 30 years we reduce the work week by a day or two for most people, compensate for our education system's decline, and avoid energy and food crises, and nothing else fundamentally changes
I can’t see us going from Microsoft and Open AI stealing everyone’s work and selling it without attribution or respect for GPL (for example ) to technological utopia anytime soon.
Especially given the average prognosis of technological progress north of a certain point is dystopia. It's like that thing where you ask everyone how many jellybeans there are in the jar and when you average the guesses the average guess is accurate. Except in this case on a societal level you average the guesses and they come out as "dystopia" but the underlying distribution contains plenty of wildly inaccurate guesses of "eutopia".
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#94The last gen of popular LLMs focuses on publicly accessible web text. But we have lots of other sources of "latent" or "hidden" text. For example, OpenAI's Whisper model can turn audio to text reliably. If you point Whisper at the world's podcasts, that's a whole new source of conversational text. If you point Whisper at YouTube, that's a whole new source of all sorts of text. And then there are all sorts of private…
But how much more data is required to make a big difference? Is doubling the dataset considered a dramatic improvement? Or is increasing the dataset by 10x needed?
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#95Earlier quoted context omitted.
> AI can generate as much synthetic data as we need, on demand. I don't think this is right. Can I take an untrained LLM (a neural network with random parameters), and have it start generating garbage, and then train the network to produce more of the same and then have it bootstrap itself to intelligence? Of course not. What if I train it just a little bit first? What if I train it until it produces gibberish, but d…
> Because so far we've never seen an AI improve based on its own output. Maybe it's because AI is such an overloaded term, but this is pretty commonplace for (semi-)supervised learning algorithms. Pseudo-labeling [1,2] is an example of this that has been around for decades. When done properly it does improve the performance of the original model, up to a certain limit (far from the singularity). Moreover, it is appar…
Boosting is reminiscent of the wisdom of the crowd effect.
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#96The last gen of popular LLMs focuses on publicly accessible web text. But we have lots of other sources of "latent" or "hidden" text. For example, OpenAI's Whisper model can turn audio to text reliably. If you point Whisper at the world's podcasts, that's a whole new source of conversational text. If you point Whisper at YouTube, that's a whole new source of all sorts of text. And then there are all sorts of private…
But how much more data is required to make a big difference? Is doubling the dataset considered a dramatic improvement? Or is increasing the dataset by 10x needed?
Also quality is likely important will the models get better if we train them on YouTube comments.
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#97I wonder if the better question is not how we get more training data but: If we're running out of training data with hallucinations and performance remaining so inadequate (per OpenAI's whitepaper) is an autoregressive transformer the right architecture? Perhaps ongoing work in finetuning will take these models to the next level but ignoring the LLM hype it really does seem like things have plateaued for a while now…
There is still an order of magnitude more organic text. Ilya Sutskever recently said it was still ok. After that, we got to use reinforcement learning (agent GPTs with tools) to generate and self-validate more examples. One "simple" application would be to build a full index of facts in the whole training corpus. Just pass each document to GPT and ask it to extract the facts. Then create an inverted index, with each…
I think the model is almost ready for real-world learning, for example in programming - sure you can have a knowledge base built on documentation, public code, etc.
But at some point you can just give the model access to tools, tell it to solve some problems, build plans, generate logs of each approach and train on those outputs. Programming is ripe for this - all the tools are easily accessible to a digital actor, everything is suited to text based model, there's plenty of tooling to provide feedback and explanations for errors geared towards humans.
No need to fumble with robotics and physical world - you can create a superhuman programmer. Then make it build out the infrastructure for physical world learning. AGI apocalypse here we come !
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#98The last gen of popular LLMs focuses on publicly accessible web text. But we have lots of other sources of "latent" or "hidden" text. For example, OpenAI's Whisper model can turn audio to text reliably. If you point Whisper at the world's podcasts, that's a whole new source of conversational text. If you point Whisper at YouTube, that's a whole new source of all sorts of text. And then there are all sorts of private…
> If you point Whisper at YouTube, that's a whole new source of all sorts of text.
a lot of YT videos already has autogenerated english subtitles, which is actually available as a vtt download, so don't even need to use Whisper on a video to obtain it!
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#99Earlier quoted context omitted.
Assuming this happens without any violence, which I truly doubt. Huge socioeconomic changes like this always come as a result of violence and uproar.
Why? Eg in the latter half of the 20th century the US integrated women into the workforce (almost doubling the population eligible for participation in the labourforce), without violence or uproar. There was also remarkably little uproar nor violence when the Czech Republic escaped the Iron Curtain and embraced capitalism.
Integrating women into the workforce is not even close to abolishing private capital and the need to work.
The violence would be between billionaires with infinite automation making infinite money, and common folk with no way to eat.
Re: Will we run out of ML data? Evidence from projecting dataset size trends (2022)
#100Earlier quoted context omitted.
I think you are mixing up layers of abstraction. The network is most likely trained with something like a categorical cross entropy loss function. Those totally punish being wrong a lot more than saying "I don't know". See https://www.v7labs.com/blog/cross-entropy-loss-guide It's just that saying "I don't know" means that your model is spreading the probability of what the next token in the text stream might be over…
Sorry if I was unclear. I know that the model is incentivised to accurately predict the probability distribution of the next token. I mean that the model is not being incentivised to literally produce the output tokens corresponding to "I don't know" when asked a question where it is uncertain.
Yes, exactly.
What I wanted to emphasize is that the training _does_ actually incentivize the model to say "I don't know" but on a lower level.