Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Neural Nets: 33 years ago and 33 years from now (2022)

karpathy.github.io

Post reply on HN
Print view View original

41–50 of 97 posts

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#41

Post by golol »

ilaksh wrote: post #34

Maybe a better idea of the future is to look at what LeCun is working on now as a future program. He wants to change quite a lot in order to move towards more animal-like cognitive abilities. LeCun is not even really interested in supervised learning anymore, for example. https://youtu.be/vyqXLJsmsrk?si=8n0ylC6qdLX06CmY Note that the talk is not really primarily about ChatGPT even though that's in the title. The new…

I don't understand why Yann is so focused on his "Animals are smarter than AI" analogy. If compute wasn't so limited, couldn't we just train a transformer on video, audio and text data? I don't see why it would not learn the basic physical structure of our world just like a language transformer learns the grammatical and other structures of language. Then with this prettaines transformer you can build an agent and use some reinforcement learning too. I feel very confident that this would mirror the level of intelligence of non-human animals quite well.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#42

Post by eru »

visarga wrote: post #22
Nevermark wrote: post #10

The most fundamental change is the difference in what models are being trained on. Little images of characters is a trivia type problem, very different from training on the linguistic and visual communication of essentially the whole human race. Another 33 years of expanded computing resources won’t be training models to mimic the behavior and knowledge of humanity. That problem ( us! ) will have been reduced to a to…

I think AI models will evolve by generating synthetic data, filtering and improving it, and then retraining. Possibly with external systems in the loop - code execution, search, human, simulation or robot. Quality won't degrade because there will be a lot of effort put into data filtering and diversity. We can always improve on a model by giving it more time. Model architecture doesn't matter compared to the dataset.…

> Model architecture doesn't matter compared to the dataset. Any model from a class can learn the same skills from the same data, but change the data and they all change their abilities - the intelligence is in the data.

You might be right in the same sense that big-O notation is 'right'. Constant factor can matter; especially once you have to take energy use into account.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#43

Post by djantje »

fnord77 wrote: post #12

> Our datasets and models today [2055] look like a joke. Both are somewhere around 10,000,000X larger. will there really be 10 million times 400 million images floating around then?

Maybe, but the input in 2055 will be more something in the form of continuous/realtime data input streams.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#44

Post by quickthrower2 »

fouronnes3 wrote: post #19
mk_stjames wrote: post #18

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of e…

This is very interesting, because I've always thought that all NN performance should be measured in a unit with energy in the denominator.

For inference that could be useful, but the energy is not for the model it is for at least the tuple of: model, model architecture and compilation, and hardware chosen.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#45

Post by jerpint »

golol wrote: post #41
ilaksh wrote: post #34

Maybe a better idea of the future is to look at what LeCun is working on now as a future program. He wants to change quite a lot in order to move towards more animal-like cognitive abilities. LeCun is not even really interested in supervised learning anymore, for example. https://youtu.be/vyqXLJsmsrk?si=8n0ylC6qdLX06CmY Note that the talk is not really primarily about ChatGPT even though that's in the title. The new…

I don't understand why Yann is so focused on his "Animals are smarter than AI" analogy. If compute wasn't so limited, couldn't we just train a transformer on video, audio and text data? I don't see why it would not learn the basic physical structure of our world just like a language transformer learns the grammatical and other structures of language. Then with this prettaines transformer you can build an agent and us…

There have been many attempts to do multi modal pre training, the difficulty is finding the right combination of data for it to be “useful” and “scalable”. It’s not trivial to just train a transformer on video, text, audio, etc. mainly due to O(N^2) on the token counts, time components with video, etc

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#46

Post by quickthrower2 »

golol wrote: post #41
ilaksh wrote: post #34

Maybe a better idea of the future is to look at what LeCun is working on now as a future program. He wants to change quite a lot in order to move towards more animal-like cognitive abilities. LeCun is not even really interested in supervised learning anymore, for example. https://youtu.be/vyqXLJsmsrk?si=8n0ylC6qdLX06CmY Note that the talk is not really primarily about ChatGPT even though that's in the title. The new…

I don't understand why Yann is so focused on his "Animals are smarter than AI" analogy. If compute wasn't so limited, couldn't we just train a transformer on video, audio and text data? I don't see why it would not learn the basic physical structure of our world just like a language transformer learns the grammatical and other structures of language. Then with this prettaines transformer you can build an agent and us…

I know intuition is often wrong, but to me a mix of dedicated visual processing AI and language AI and sound AI all somehow interacting with each other would be a good way to make an “animal” like AI rather than throw loads of attention heads at everything all at once.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#47

Post by quickthrower2 »

retrac wrote: post #8
version_five wrote: post #2

This was really good. The only thing I didn't see explicitly discussed, although I guess it's obvious, is that what's different 33 years later is the inputs the models operate on. The '89 sota model used 16x16 greyscale images, today we have single digit megapixel color images, in 30 years, a desktop will be able to train Clip in 90 seconds, but what will the sota models be trained on?

Human behaviour in a way far more general than which token we might next type. To mimick humans as closely as might be possible with the basic deep learning method, train something that can predict human behaviour in general. Training would require billions to quadrillions of hours of video and audio and probably many other inputs, from many different people, engaged in the full variety of human activity.

Humans have a brain that physically changes especially in childhood, so that is potentially a massive advantage.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#48

Post by georgeg23 »

mkaic wrote: post #36
canjobear wrote: post #31

Earlier quoted context omitted.

Neural nets seem to learn much slower than humans. Even GPT-2 has seen orders of magnitude more tokens of language than a human experiences in a lifetime. At least as far as language is concerned, humans are able to extract a lot more information from their training data.

Humans are also extensively pretrained by billions of years of evolution, so by starting from scratch GPT is admittedly disadvantaged from the get-go.

But the human genome is only 3 gigabytes and the vast majority of that is unlikely to be encoding brain structure.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#49

Post by swores »

jacquesm wrote: post #37
swores wrote: post #33

Earlier quoted context omitted.

My (wrong) intuition on reading your comment was that you were over-estimating the expected growth in performance over that time period, but actually after checking the maths based on Moore's Law, i.e. doubling every two years (though of course I understand that was a rough estimate, more of a concept prediction than expected to be precise) you're right so I'll share the maths for anyone else whose intuition might be…

The rest of that difference can easily be explained by the difference in the class of hardware used. A desktop made today vs a laptop is roughly that factor 4. Not sure if back then there would have been laptops that you could have done this on for a more apples-to-apples comparison. Modern laptops give great efficiency, when I went for solar power here the first thing to go was the desktop computer. I still have it,…

Hadn't thought of that, good point

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#50

Post by ResearchCode »

Nevermark wrote: post #10

The most fundamental change is the difference in what models are being trained on. Little images of characters is a trivia type problem, very different from training on the linguistic and visual communication of essentially the whole human race. Another 33 years of expanded computing resources won’t be training models to mimic the behavior and knowledge of humanity. That problem ( us! ) will have been reduced to a to…

Come close to solving the toy problem of autonomous driving first, we're still waiting.
Post reply on HN