> Our datasets and models today [2055] look like a joke. Both are somewhere around 10,000,000X larger. will there really be 10 million times 400 million images floating around then?
No, there won't. I must assume he is exaggerating for the clicks.
Deep Neural Nets: 33 years ago and 33 years from now (2022)
91–97 of 97 posts
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#92Earlier quoted context omitted.
It totally depends on what you want to use a measure for. Just like neither height or volume alone will tell you what will fit in your car. By any measure that puts energy used by the brain in the denominator, humans are probably dumber than ants. But that doesn't mean those measures are always accurate. (For contemporary neural networks, you also have to distinguish training costs from inference costs.)
To add more context, humans are 100W biological machines. Brain is ~20% of that power - 20W. The greatest form of general intelligence at 20W. A MacBook Air is ~30W. https://www.jackery.com/blogs/knowledge/how-many-watts-a-lap...
That’s not entirely fair. That’s run time cost but not the same cost to information ratio and the cost of drawing from that pool.
Given a laptop is at 30W how much can a laptop do disconnected from the internet? Now how much can it do with the internet? Now how much information does the internet cost in terms of wattage? Now what’s the ratio?
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#93Earlier quoted context omitted.
Why? An adult by 25 only has 146k hours of video experience “training,” most of it repeated, derivative, and unproductive. And their encoded genes can be observed in their genome, so don’t need to be retrained by millions of years of evolution.
Neural nets seem to learn much slower than humans. Even GPT-2 has seen orders of magnitude more tokens of language than a human experiences in a lifetime. At least as far as language is concerned, humans are able to extract a lot more information from their training data.
One difference is that humans are actively involved in data collection, so when there is a gap in their knowledge, they don’t just wait for the information to show up, they ask a question, etc.
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#94Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#95Earlier quoted context omitted.
To add more context, humans are 100W biological machines. Brain is ~20% of that power - 20W. The greatest form of general intelligence at 20W. A MacBook Air is ~30W. https://www.jackery.com/blogs/knowledge/how-many-watts-a-lap...
That’s not entirely fair. That’s run time cost but not the same cost to information ratio and the cost of drawing from that pool. Given a laptop is at 30W how much can a laptop do disconnected from the internet? Now how much can it do with the internet? Now how much information does the internet cost in terms of wattage? Now what’s the ratio?
What can a human do disconnected from society?
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#96Earlier quoted context omitted.
I don't understand why Yann is so focused on his "Animals are smarter than AI" analogy. If compute wasn't so limited, couldn't we just train a transformer on video, audio and text data? I don't see why it would not learn the basic physical structure of our world just like a language transformer learns the grammatical and other structures of language. Then with this prettaines transformer you can build an agent and us…
There have been many attempts to do multi modal pre training, the difficulty is finding the right combination of data for it to be “useful” and “scalable”. It’s not trivial to just train a transformer on video, text, audio, etc. mainly due to O(N^2) on the token counts, time components with video, etc
I don't really see it. O(N^2) for context window length is not an issue as you don't need particularly much longer context windows than for text. You don't have to run this at 30 FPS, 12 would already be enough to understand what's going on. The dimensionality is much higher of course, but how many dimensions does the latent space of TikTok videos really have? Train an autoencoder and take only 1000 dimensions for the frames. Of course it's not that simple but my mental models atm make me feel like this should work. What do I mean by work? A grainy low fps video+subtitles transformer that looks terrible but as a plus actually has some decent physical consistency. I guess that is Phenaki etc., but I'd hope for much better understanding if the "grammar" of the physical world. That should be rather orthogonal to FPS or resolution (or number of dimensions you take).
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#97Earlier quoted context omitted.
I don't understand why Yann is so focused on his "Animals are smarter than AI" analogy. If compute wasn't so limited, couldn't we just train a transformer on video, audio and text data? I don't see why it would not learn the basic physical structure of our world just like a language transformer learns the grammatical and other structures of language. Then with this prettaines transformer you can build an agent and us…
I know intuition is often wrong, but to me a mix of dedicated visual processing AI and language AI and sound AI all somehow interacting with each other would be a good way to make an “animal” like AI rather than throw loads of attention heads at everything all at once.
I'm just thinking of google's GATO, it seemed to have absolutely no trouble integrating a large number of modes.