Deep Neural Nets: 33 years ago and 33 years from now (2022)
karpathy.github.io
Deep Neural Nets: 33 years ago and 33 years from now (2022)
1–10 of 97 posts
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#2This was really good. The only thing I didn't see explicitly discussed, although I guess it's obvious, is that what's different 33 years later is the inputs the models operate on. The '89 sota model used 16x16 greyscale images, today we have single digit megapixel color images, in 30 years, a desktop will be able to train Clip in 90 seconds, but what will the sota models be trained on?
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#3It's crazy how little has changed and how much had changed. I remember what a revelation "the unreasonable effectiveness of RNNs" was when I was read it and it feels like we live in a different world.
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#4Related:
Deep Neural Nets: 33 years ago and 33 years from now - https://news.ycombinator.com/item?id=30673821 - March 2022 (5 comments)
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#5It’s interesting in that time we almost completely lost interest in neural networks and then came back around to them.
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#6It's crazy how little has changed and how much had changed. I remember what a revelation "the unreasonable effectiveness of RNNs" was when I was read it and it feels like we live in a different world.
I think we could collectively more constructive and sober conversation if we kept that 2015 bit of work as a sort of baseline.
The new stuff is better, by a lot, and with implications more to come.
But those of us paying attention then had a frame of reference where “so much better it’s crazy” still stops short of “it’s out of control”.
It’s a lot better.
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#7[flagged]
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#8This was really good. The only thing I didn't see explicitly discussed, although I guess it's obvious, is that what's different 33 years later is the inputs the models operate on. The '89 sota model used 16x16 greyscale images, today we have single digit megapixel color images, in 30 years, a desktop will be able to train Clip in 90 seconds, but what will the sota models be trained on?
Human behaviour in a way far more general than which token we might next type. To mimick humans as closely as might be possible with the basic deep learning method, train something that can predict human behaviour in general. Training would require billions to quadrillions of hours of video and audio and probably many other inputs, from many different people, engaged in the full variety of human activity.
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#9This was really good. The only thing I didn't see explicitly discussed, although I guess it's obvious, is that what's different 33 years later is the inputs the models operate on. The '89 sota model used 16x16 greyscale images, today we have single digit megapixel color images, in 30 years, a desktop will be able to train Clip in 90 seconds, but what will the sota models be trained on?
Human behaviour in a way far more general than which token we might next type. To mimick humans as closely as might be possible with the basic deep learning method, train something that can predict human behaviour in general. Training would require billions to quadrillions of hours of video and audio and probably many other inputs, from many different people, engaged in the full variety of human activity.
Why? An adult by 25 only has 146k hours of video experience “training,” most of it repeated, derivative, and unproductive. And their encoded genes can be observed in their genome, so don’t need to be retrained by millions of years of evolution.
Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)
#10The most fundamental change is the difference in what models are being trained on.
Little images of characters is a trivia type problem, very different from training on the linguistic and visual communication of essentially the whole human race.
Another 33 years of expanded computing resources won’t be training models to mimic the behavior and knowledge of humanity.
That problem (us!) will have been reduced to a toy problem long before then.