Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Neural Nets: 33 years ago and 33 years from now (2022)

karpathy.github.io

Post reply on HN
Print view View original

61–70 of 97 posts

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#61

Post by retrac »

georgeg23 wrote: post #48
mkaic wrote: post #36

Earlier quoted context omitted.

Humans are also extensively pretrained by billions of years of evolution, so by starting from scratch GPT is admittedly disadvantaged from the get-go.

But the human genome is only 3 gigabytes and the vast majority of that is unlikely to be encoding brain structure.

"Only" 3 gigabytes.

The lambda calculus (a system we know is capable of infinite self-complexity, learning, etc. with the right program) can be described in a few hundred bits. And a neural net can be described in the lambda calculus in perhaps a few thousand bits.

Also, we have no idea how "compressed" the genome is.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#62

Post by breckinloggins »

I really enjoyed this article. My only critique is that the 2055 predictions are "meta-linear". In other words: the author avoids the (probable) mistake of taking our current tech and linearly regressing the numbers 33 years forward, but the predictions still suggest a kind of "worldline symmetry" with the present date at the origin.

It's quite possible that none of these predictions will come true simply because the timeframe is large enough for many unanticipated breakthroughs and roadblocks.

Maybe someone will figure out a much, much simpler foundational architecture than "perceptrons++", maybe we'll all be training clouds of 3D gaussians, maybe quantum computers will finally take off and we don't even have the nouns for the building blocks we'll use.

On the negative side perhaps we hit a hard scaling limit (in hardware or training) that we didn't see coming. Or a civilizational setback.

All that said, though, if I were a betting man I wouldn't exactly wager against the article's conclusions; they're probably the best we can extrapolate knowing only the past and present state of affairs.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#63

Post by canjobear »

mkaic wrote: post #36
canjobear wrote: post #31

Earlier quoted context omitted.

Neural nets seem to learn much slower than humans. Even GPT-2 has seen orders of magnitude more tokens of language than a human experiences in a lifetime. At least as far as language is concerned, humans are able to extract a lot more information from their training data.

Humans are also extensively pretrained by billions of years of evolution, so by starting from scratch GPT is admittedly disadvantaged from the get-go.

True, but we don’t know how to recreate that kind of pretraining.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#64

Post by mark_l_watson »

Great article. I lived through the early days of artificial neural networks. I was on a DARPA advisory panel for neural network tooling in the mid 1980s, wrote the first version of the SAIC ANSim commercial product, and created the simple back-prop model that was deployed in the bomb detector my company built under contract to the FAA. I also managed a ‘conventional’ deep learning team at Capital One 5-6 years ago.

My world has been very exciting in the last 18 months. I spend as much time as I can exploring self hosted LLMs, APIs from Hugging Face, OpenAI, etc.

My mind is blown even thinking about tech 33 years from now!

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#65

Post by Nevermark »

ResearchCode wrote: post #50
Nevermark wrote: post #10

The most fundamental change is the difference in what models are being trained on. Little images of characters is a trivia type problem, very different from training on the linguistic and visual communication of essentially the whole human race. Another 33 years of expanded computing resources won’t be training models to mimic the behavior and knowledge of humanity. That problem ( us! ) will have been reduced to a to…

Come close to solving the toy problem of autonomous driving first, we're still waiting.

I don’t know. I find pessimistic views, like you are expressing, very strange.

My Tesla drives and navigates itself most of the time. 90-95% at least, just not 100%.

As apposed to cars 10 or more years ago which didn’t do any of that.

To me it is much like the “God of the Gaps” when tremendous progress on a big problem is dismissed negatively, due to the (continuously shrinking) gaps of what it can’t do.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#66

Post by chubot »

eru wrote: post #40
fouronnes3 wrote: post #19

Earlier quoted context omitted.

This is very interesting, because I've always thought that all NN performance should be measured in a unit with energy in the denominator.

It totally depends on what you want to use a measure for. Just like neither height or volume alone will tell you what will fit in your car. By any measure that puts energy used by the brain in the denominator, humans are probably dumber than ants. But that doesn't mean those measures are always accurate. (For contemporary neural networks, you also have to distinguish training costs from inference costs.)

I dispute that, if the metric is a chess game between an ant and a human

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#67

Post by fooker »

Aardwolf wrote: post #25
mk_stjames wrote: post #18

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of e…

30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then

Amdahl's law

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#68

Post by Yajirobe »

mk_stjames wrote: post #18

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of e…

> watt-hours

You mean joules (up to a constant factor)?

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#69

Post by renonce »

nojvek wrote: post #56
eru wrote: post #40

Earlier quoted context omitted.

It totally depends on what you want to use a measure for. Just like neither height or volume alone will tell you what will fit in your car. By any measure that puts energy used by the brain in the denominator, humans are probably dumber than ants. But that doesn't mean those measures are always accurate. (For contemporary neural networks, you also have to distinguish training costs from inference costs.)

To add more context, humans are 100W biological machines. Brain is ~20% of that power - 20W. The greatest form of general intelligence at 20W. A MacBook Air is ~30W. https://www.jackery.com/blogs/knowledge/how-many-watts-a-lap...

It is “the greatest” because we only appreciate intelligence that we ourselves understand. A 0.0001W calculator calculates arithmetic faster than any human brain.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#70

Post by renonce »

Yajirobe wrote: post #68
mk_stjames wrote: post #18

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of e…

> watt-hours You mean joules (up to a constant factor)?

A watt-hour is 3600 joules but watt-hours or kilowatt-hours is commonly used because it’s easier to calculate.
Post reply on HN