Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Neural Nets: 33 years ago and 33 years from now (2022)

karpathy.github.io

Post reply on HN
Print view View original

31–40 of 97 posts

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#31

Post by canjobear »

reducesuffering wrote: post #9
retrac wrote: post #8

Earlier quoted context omitted.

Human behaviour in a way far more general than which token we might next type. To mimick humans as closely as might be possible with the basic deep learning method, train something that can predict human behaviour in general. Training would require billions to quadrillions of hours of video and audio and probably many other inputs, from many different people, engaged in the full variety of human activity.

Why? An adult by 25 only has 146k hours of video experience “training,” most of it repeated, derivative, and unproductive. And their encoded genes can be observed in their genome, so don’t need to be retrained by millions of years of evolution.

Neural nets seem to learn much slower than humans. Even GPT-2 has seen orders of magnitude more tokens of language than a human experiences in a lifetime. At least as far as language is concerned, humans are able to extract a lot more information from their training data.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#32

Post by somewhereoutth »

dsign wrote: post #30

It's not clear that compute will scale as it did for the next 33 years. But it doesn't really need to. I read the article and I was thinking "my God, I remember I used MSE that weekend in my pet ML project and it really didn't work out that well; wrong loss function." Our current crop of LLMs, or the one next year, will be perfectly able to tell me how I can improve my code and graphs, which means that I can deploy s…

> or the one next year

We have heard, and will continue to hear, this sort of thing rather a lot. The last 5 yards are the hardest, but without them the previous 5 miles are of limited utility.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#33

Post by swores »

Aardwolf wrote: post #25
mk_stjames wrote: post #18

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of e…

30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then

My (wrong) intuition on reading your comment was that you were over-estimating the expected growth in performance over that time period, but actually after checking the maths based on Moore's Law, i.e. doubling every two years (though of course I understand that was a rough estimate, more of a concept prediction than expected to be precise) you're right so I'll share the maths for anyone else whose intuition might be as poor as mine:

Doubling every 2 years = compound annual growth rate (CAGR) of ~41.42%

  CAGR = ((End Value / Start Value)^(1 / Number of Years)) - 1

  ((2 / 1)^(1 / 2)) - 1 = 0.41421356237
Therefore in 34 years since then:

  1 * (1 + 0.41421356237)^34 = ~131,072
So x30k is ~4.4x less than 131k. Then again, that's equivalent to ~x1.833 every two years, compared to Moore's Law of x2 every two years, so only ~8% less growth per two years, which coming back to the fact that Moore's Law is a rough estimate concept not an exact fact, doesn't seem to far off!

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#34

Post by ilaksh »

Maybe a better idea of the future is to look at what LeCun is working on now as a future program. He wants to change quite a lot in order to move towards more animal-like cognitive abilities.

LeCun is not even really interested in supervised learning anymore, for example.

https://youtu.be/vyqXLJsmsrk?si=8n0ylC6qdLX06CmY

Note that the talk is not really primarily about ChatGPT even though that's in the title. The new ideas are a little bit in. The beginning of the talk is just him explaining how unimpressed he is with LLMs. Which I think is a misjudgement but that doesn't mean his plan doesn't have merit.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#35

Post by anonzzzies »

ben_w wrote: post #28
fnord77 wrote: post #12

> Our datasets and models today [2055] look like a joke. Both are somewhere around 10,000,000X larger. will there really be 10 million times 400 million images floating around then?

10e6*400e6/8e9/365/18 = 76 images per person per waking hour; it's not implausible given how many cameras there are and how many moments people might snap to share with remote friends — I can easily believe we'll have always-on video chat with multiple people in AR glasses by that point.

Most images are not shared though; just snapped. In the past you had photo albums no-one ever looked in. And that weren't that many pics; now , whenever, people (old and young) take 100s of pictures, on iPhones often by holding the button so it zaps 100s of them in a few seconds.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#36

Post by mkaic »

canjobear wrote: post #31
reducesuffering wrote: post #9

Earlier quoted context omitted.

Why? An adult by 25 only has 146k hours of video experience “training,” most of it repeated, derivative, and unproductive. And their encoded genes can be observed in their genome, so don’t need to be retrained by millions of years of evolution.

Neural nets seem to learn much slower than humans. Even GPT-2 has seen orders of magnitude more tokens of language than a human experiences in a lifetime. At least as far as language is concerned, humans are able to extract a lot more information from their training data.

Humans are also extensively pretrained by billions of years of evolution, so by starting from scratch GPT is admittedly disadvantaged from the get-go.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#37

Post by jacquesm »

swores wrote: post #33
Aardwolf wrote: post #25

Earlier quoted context omitted.

30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then

My (wrong) intuition on reading your comment was that you were over-estimating the expected growth in performance over that time period, but actually after checking the maths based on Moore's Law, i.e. doubling every two years (though of course I understand that was a rough estimate, more of a concept prediction than expected to be precise) you're right so I'll share the maths for anyone else whose intuition might be…

The rest of that difference can easily be explained by the difference in the class of hardware used. A desktop made today vs a laptop is roughly that factor 4. Not sure if back then there would have been laptops that you could have done this on for a more apples-to-apples comparison.

Modern laptops give great efficiency, when I went for solar power here the first thing to go was the desktop computer. I still have it, but it hasn't run in over a year and the elderly thinkpad that is now my daily driver uses far less power and still has enough compute to serve my modest needs. But if I would dive into something requiring much more compute I'd have to start the desktop again. Unfortunately power management is not such that computers can really throttle down to 'miser mode' when you don't need it, it's a good step but not as good as the jump between desktop and laptop.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#38

Post by ben_w »

anonzzzies wrote: post #35
ben_w wrote: post #28

Earlier quoted context omitted.

10e6*400e6/8e9/365/18 = 76 images per person per waking hour; it's not implausible given how many cameras there are and how many moments people might snap to share with remote friends — I can easily believe we'll have always-on video chat with multiple people in AR glasses by that point.

Most images are not shared though; just snapped. In the past you had photo albums no-one ever looked in. And that weren't that many pics; now , whenever, people (old and young) take 100s of pictures, on iPhones often by holding the button so it zaps 100s of them in a few seconds.

> Most images are not shared though

Not yet.

As the joke goes:

People in the 60s:

I better not say that or the government will wiretap my house

People today:

Hey wiretap, do you have a recipe for pancakes?

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#39

Post by jacquesm »

bilsbie wrote: post #5

It’s interesting in that time we almost completely lost interest in neural networks and then came back around to them.

Thank Hinton for that. It's a pity we don't have a Nobel for software.

But a Turing award is pretty neat as well.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#40

Post by eru »

fouronnes3 wrote: post #19
mk_stjames wrote: post #18

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of e…

This is very interesting, because I've always thought that all NN performance should be measured in a unit with energy in the denominator.

It totally depends on what you want to use a measure for. Just like neither height or volume alone will tell you what will fit in your car.

By any measure that puts energy used by the brain in the denominator, humans are probably dumber than ants. But that doesn't mean those measures are always accurate.

(For contemporary neural networks, you also have to distinguish training costs from inference costs.)

Post reply on HN