Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Neural Nets: 33 years ago and 33 years from now (2022)

karpathy.github.io

Post reply on HN
Print view View original

11–20 of 97 posts

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#11

Post by [deleted] »

reducesuffering wrote: post #9
retrac wrote: post #8

Earlier quoted context omitted.

Human behaviour in a way far more general than which token we might next type. To mimick humans as closely as might be possible with the basic deep learning method, train something that can predict human behaviour in general. Training would require billions to quadrillions of hours of video and audio and probably many other inputs, from many different people, engaged in the full variety of human activity.

Why? An adult by 25 only has 146k hours of video experience “training,” most of it repeated, derivative, and unproductive. And their encoded genes can be observed in their genome, so don’t need to be retrained by millions of years of evolution.

[deleted]

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#13

Post by alanbernstein »

reducesuffering wrote: post #9
retrac wrote: post #8

Earlier quoted context omitted.

Human behaviour in a way far more general than which token we might next type. To mimick humans as closely as might be possible with the basic deep learning method, train something that can predict human behaviour in general. Training would require billions to quadrillions of hours of video and audio and probably many other inputs, from many different people, engaged in the full variety of human activity.

Why? An adult by 25 only has 146k hours of video experience “training,” most of it repeated, derivative, and unproductive. And their encoded genes can be observed in their genome, so don’t need to be retrained by millions of years of evolution.

How better to learn to do menial physical tasks like house cleaning, and produce picking?

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#14

Post by thaw13579 »

reducesuffering wrote: post #9
retrac wrote: post #8

Earlier quoted context omitted.

Human behaviour in a way far more general than which token we might next type. To mimick humans as closely as might be possible with the basic deep learning method, train something that can predict human behaviour in general. Training would require billions to quadrillions of hours of video and audio and probably many other inputs, from many different people, engaged in the full variety of human activity.

Why? An adult by 25 only has 146k hours of video experience “training,” most of it repeated, derivative, and unproductive. And their encoded genes can be observed in their genome, so don’t need to be retrained by millions of years of evolution.

Much of that time also includes physical interaction with the world, which makes it far more valuable because it can improve performance in a focused way.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#15

Post by sroussey »

bilsbie wrote: post #5

It’s interesting in that time we almost completely lost interest in neural networks and then came back around to them.

I had to retake my AI class at university several times because I just didn’t agree on the “AI is symbolic search” aspect.

Now though, I’m sure people are taking LLMs and putting them together to do forward and backward chaining.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#17

Post by sroussey »

sroussey wrote: post #16
fnord77 wrote: post #12

> Our datasets and models today [2055] look like a joke. Both are somewhere around 10,000,000X larger. will there really be 10 million times 400 million images floating around then?

Generate as many as you need.

Oh, also curious… today, how many individual image frames from video are there just from Tesla vehicles?

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#18

Post by mk_stjames »

Something else I find exciting, starting with one of the reflections-

The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that).

So 200 watts * 72 hours = 14400 watt-hours of energy.

Karpathy trained the equivalent on a Macbook, not even fully utilized, in 90 seconds. Likely something around 20 watts * 0.025 hours = 0.5 watt-hours.

An energy efficiency improvement of nearly 30000x.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#19

Post by fouronnes3 »

mk_stjames wrote: post #18

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of e…

This is very interesting, because I've always thought that all NN performance should be measured in a unit with energy in the denominator.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#20

Post by alpaca128 »

sroussey wrote: post #16
fnord77 wrote: post #12

> Our datasets and models today [2055] look like a joke. Both are somewhere around 10,000,000X larger. will there really be 10 million times 400 million images floating around then?

Generate as many as you need.

Training models from generated content degrades them over time.
Post reply on HN