Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Neural Nets: 33 years ago and 33 years from now (2022)

karpathy.github.io

Post reply on HN
Print view View original

71–80 of 97 posts

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#71

Post by cma »

Aardwolf wrote: post #25
mk_stjames wrote: post #18

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of e…

30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then

33 years ago is 2000/1999

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#72

Post by Philpax »

cma wrote: post #71
Aardwolf wrote: post #25

Earlier quoted context omitted.

30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then

33 years ago is 2000/1999

Um... you might want to check your tens digit.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#73

Post by Philpax »

alpaca128 wrote: post #20
sroussey wrote: post #16

Earlier quoted context omitted.

Generate as many as you need.

Training models from generated content degrades them over time.

The generated results can come from other means - for example, pretraining on rendered CG imagery is quite popular in the computer vision world, especially for problems where acquiring ground truth data in the real world is quite difficult.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#74

Post by cma »

Nevermark wrote: post #65
ResearchCode wrote: post #50

Earlier quoted context omitted.

Come close to solving the toy problem of autonomous driving first, we're still waiting.

I don’t know. I find pessimistic views, like you are expressing, very strange. My Tesla drives and navigates itself most of the time. 90-95% at least, just not 100%. As apposed to cars 10 or more years ago which didn’t do any of that. To me it is much like the “God of the Gaps” when tremendous progress on a big problem is dismissed negatively, due to the (continuously shrinking) gaps of what it can’t do.

Waymo and Cruise are operating without drivers which is much more impressive, even if in limited areas.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#75

Post by cma »

anonzzzies wrote: post #35
ben_w wrote: post #28

Earlier quoted context omitted.

10e6*400e6/8e9/365/18 = 76 images per person per waking hour; it's not implausible given how many cameras there are and how many moments people might snap to share with remote friends — I can easily believe we'll have always-on video chat with multiple people in AR glasses by that point.

Most images are not shared though; just snapped. In the past you had photo albums no-one ever looked in. And that weren't that many pics; now , whenever, people (old and young) take 100s of pictures, on iPhones often by holding the button so it zaps 100s of them in a few seconds.

Maybe you won't receive your "world coin" universal income dividend unless you livestream 24/7.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#76

Post by 0xDEF »

phkahler wrote: post #60

>> The original network trained for 3 days on a SUN-4/260 workstation. This is exactly why I didn't start experimenting with this stuff back then. I read some articles and had the interest, but having no access to existing training data or "fast" computers was really a show stopper. This article really convinced me that the amazing results today are mostly due to hardware advances. I will add my own view that 1) hard…

>This article really convinced me that the amazing results today are mostly due to hardware advances.

For LLMs that is true. But many other things like Whisper, Stable Diffusion etc. could in theory have been made a decade earlier.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#77

Post by georgeg23 »

retrac wrote: post #61
georgeg23 wrote: post #48

Earlier quoted context omitted.

But the human genome is only 3 gigabytes and the vast majority of that is unlikely to be encoding brain structure.

"Only" 3 gigabytes. The lambda calculus (a system we know is capable of infinite self-complexity, learning, etc. with the right program) can be described in a few hundred bits. And a neural net can be described in the lambda calculus in perhaps a few thousand bits. Also, we have no idea how "compressed" the genome is.

Basic structure can be encoded yes (and obviously is given brains have consistent structure), but the weights or parameters, presuming that brains learn via synaptic weights, obviously do not fit in the genome.

Compression still must obey information theory.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#78

Post by Nevermark »

cma wrote: post #74
Nevermark wrote: post #65

Earlier quoted context omitted.

I don’t know. I find pessimistic views, like you are expressing, very strange. My Tesla drives and navigates itself most of the time. 90-95% at least, just not 100%. As apposed to cars 10 or more years ago which didn’t do any of that. To me it is much like the “God of the Gaps” when tremendous progress on a big problem is dismissed negatively, due to the (continuously shrinking) gaps of what it can’t do.

Waymo and Cruise are operating without drivers which is much more impressive, even if in limited areas.

I don’t know if it is much more impressive than Tesla, given Tesla’s “limited areas” don’t seem very limiting in my experience.

But I think all these complementary takes on the problem, with significant year-to-year progress by all three firms, are fantastic.

That used to be considered a fast learning curve!

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#79

Post by Nevermark »

breckinloggins wrote: post #62

I really enjoyed this article. My only critique is that the 2055 predictions are "meta-linear". In other words: the author avoids the (probable) mistake of taking our current tech and linearly regressing the numbers 33 years forward, but the predictions still suggest a kind of "worldline symmetry" with the present date at the origin. It's quite possible that none of these predictions will come true simply because the…

I think you are right, the next 33 years are likely to be very different.

I would lean to them being even more dramatic, due to the opportunity to advance algorithms, not just resources.

On the more obvious side, most libraries are not yet taking full advantage of many known gradient optimization techniques. It’s been so much easier to just add data & processing that there is an overhangs of tools to still apply.

And large successful models are telling us important things.

For instance, it is clear that language models are learning a kind of logic of language similar to how we process thoughts, allowing highly disparate types information to be woven together sensibly.

At some point, identifying the nature of that processing could radically simplify language processing.

That is just one opportunity for radical architecture and algorithm advances, and it would be revolutionary.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#80

Post by edgyquant »

reducesuffering wrote: post #9
retrac wrote: post #8

Earlier quoted context omitted.

Human behaviour in a way far more general than which token we might next type. To mimick humans as closely as might be possible with the basic deep learning method, train something that can predict human behaviour in general. Training would require billions to quadrillions of hours of video and audio and probably many other inputs, from many different people, engaged in the full variety of human activity.

Why? An adult by 25 only has 146k hours of video experience “training,” most of it repeated, derivative, and unproductive. And their encoded genes can be observed in their genome, so don’t need to be retrained by millions of years of evolution.

Humans do not train on video, the idea of a video, or even a frame, is a high level abstraction within the human brain.
Post reply on HN