Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Neural Nets: 33 years ago and 33 years from now (2022)

karpathy.github.io

Post reply on HN
Print view View original

21–30 of 97 posts

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#21

Post by ramblerman »

version_five wrote: post #2

This was really good. The only thing I didn't see explicitly discussed, although I guess it's obvious, is that what's different 33 years later is the inputs the models operate on. The '89 sota model used 16x16 greyscale images, today we have single digit megapixel color images, in 30 years, a desktop will be able to train Clip in 90 seconds, but what will the sota models be trained on?

Millions of hours of data captured by headsets like the vision pro?

Not sure all the things it captures, but a model could be trained on the combination of audio/video/spatial/iris/what have you...

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#22

Post by visarga »

Nevermark wrote: post #10

The most fundamental change is the difference in what models are being trained on. Little images of characters is a trivia type problem, very different from training on the linguistic and visual communication of essentially the whole human race. Another 33 years of expanded computing resources won’t be training models to mimic the behavior and knowledge of humanity. That problem ( us! ) will have been reduced to a to…

I think AI models will evolve by generating synthetic data, filtering and improving it, and then retraining. Possibly with external systems in the loop - code execution, search, human, simulation or robot. Quality won't degrade because there will be a lot of effort put into data filtering and diversity. We can always improve on a model by giving it more time.

Model architecture doesn't matter compared to the dataset. Any model from a class can learn the same skills from the same data, but change the data and they all change their abilities - the intelligence is in the data.

The future is data engineering, not model architecturing. Human culture, by analogy, evolves faster than human biology. The data is evolving faster than the model. And we are seeing a drastic reduction in novel architectures in AI, diverse datasets applied to the same transformer models in recent years. Even among the transformers, very few variants are largely used, thousands of them abandoned.

I like to think of it as language evolution by memetics being the real engine behind intelligence. We and AI are riding the language exponential together.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#23

Post by ramblerman »

fnord77 wrote: post #12

> Our datasets and models today [2055] look like a joke. Both are somewhere around 10,000,000X larger. will there really be 10 million times 400 million images floating around then?

I think you are limiting yourself by thinking of the dataset of the future as just being more and bigger images.

Perhaps it will be trained on whole videos, or a combination of different inputs from agents that move about in the real world / or a video game.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#25

Post by Aardwolf »

mk_stjames wrote: post #18

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of e…

30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#27

Post by ben_w »

Aardwolf wrote: post #25
mk_stjames wrote: post #18

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of e…

30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then

It isn't much, but as the link says, the neural network they were reimplementing is too small to take advantage of modern hardware.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#28

Post by ben_w »

fnord77 wrote: post #12

> Our datasets and models today [2055] look like a joke. Both are somewhere around 10,000,000X larger. will there really be 10 million times 400 million images floating around then?

10e6*400e6/8e9/365/18 = 76 images per person per waking hour; it's not implausible given how many cameras there are and how many moments people might snap to share with remote friends — I can easily believe we'll have always-on video chat with multiple people in AR glasses by that point.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#29

Post by Lucasoato »

ramblerman wrote: post #23
fnord77 wrote: post #12

> Our datasets and models today [2055] look like a joke. Both are somewhere around 10,000,000X larger. will there really be 10 million times 400 million images floating around then?

I think you are limiting yourself by thinking of the dataset of the future as just being more and bigger images. Perhaps it will be trained on whole videos, or a combination of different inputs from agents that move about in the real world / or a video game.

Maybe the real game changer in the future will be the ability to train the same model on very different kind of inputs like video, images, text, audio... Imagine also all these data cleaning tasks are already automated, you just need to feed the model PDFs and automatically a support model will extract all the relevant metadata... or probably you'll just be able to select a set of books from an online library and your model will train on them as well (of course for a non trivial subscription lol)

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#30

Post by dsign »

It's not clear that compute will scale as it did for the next 33 years. But it doesn't really need to.

I read the article and I was thinking "my God, I remember I used MSE that weekend in my pet ML project and it really didn't work out that well; wrong loss function." Our current crop of LLMs, or the one next year, will be perfectly able to tell me how I can improve my code and graphs, which means that I can deploy some expert-level techniques that otherwise would be "locked" to me by 50000 hours of "mastery acquisition".

A part of me is telling me that we humans are doomed, and that in 33 years we would have created a world in which we humans are irrelevant. But another part tells me that if we avoid that fate and all the other dooms, the future might just be quite bright.

Post reply on HN