Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Neural Nets: 33 years ago and 33 years from now (2022)

karpathy.github.io

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#81

Post by ResearchCode »

Nevermark wrote: post #65
ResearchCode wrote: post #50

Earlier quoted context omitted.

Come close to solving the toy problem of autonomous driving first, we're still waiting.

I don’t know. I find pessimistic views, like you are expressing, very strange. My Tesla drives and navigates itself most of the time. 90-95% at least, just not 100%. As apposed to cars 10 or more years ago which didn’t do any of that. To me it is much like the “God of the Gaps” when tremendous progress on a big problem is dismissed negatively, due to the (continuously shrinking) gaps of what it can’t do.

We are already five years late in autonomous vehicles replacing all truck drivers. We will see if we even have autonomous driving "long before" 33 years have passed. "AGI" (rebranded AI after "AI" failed to deliver) will of course still not be a thing.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#82

Post by fHr »

cma wrote: post #71
Aardwolf wrote: post #25

Earlier quoted context omitted.

30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then

33 years ago is 2000/1999

quickest maffs

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#83

Post by Nevermark »

ResearchCode wrote: post #81
Nevermark wrote: post #65

Earlier quoted context omitted.

I don’t know. I find pessimistic views, like you are expressing, very strange. My Tesla drives and navigates itself most of the time. 90-95% at least, just not 100%. As apposed to cars 10 or more years ago which didn’t do any of that. To me it is much like the “God of the Gaps” when tremendous progress on a big problem is dismissed negatively, due to the (continuously shrinking) gaps of what it can’t do.

We are already five years late in autonomous vehicles replacing all truck drivers. We will see if we even have autonomous driving "long before" 33 years have passed. "AGI" (rebranded AI after "AI" failed to deliver) will of course still not be a thing.

I have no idea what “late” technology means.

And not delivering AGI yet is a problem?

What are these broad technology schedule based criticisms founded on?

I really want to understand this viewpoint!

Hopefully not the over-optimism of anyone who uses optimistic timelines as a motivational force. That’s not real data. Or a suitable benchmark for human progress.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#84

Post by DesiLurker »

jacquesm wrote: post #37
swores wrote: post #33

Earlier quoted context omitted.

My (wrong) intuition on reading your comment was that you were over-estimating the expected growth in performance over that time period, but actually after checking the maths based on Moore's Law, i.e. doubling every two years (though of course I understand that was a rough estimate, more of a concept prediction than expected to be precise) you're right so I'll share the maths for anyone else whose intuition might be…

The rest of that difference can easily be explained by the difference in the class of hardware used. A desktop made today vs a laptop is roughly that factor 4. Not sure if back then there would have been laptops that you could have done this on for a more apples-to-apples comparison. Modern laptops give great efficiency, when I went for solar power here the first thing to go was the desktop computer. I still have it,…

also the 'memory wall', remember memory b/w did not grow at pace with moore's law. sure there are ways to mitigate it but that eat into chip budget and reflect when real world performance is calculated.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#85

Post by vendiddy »

AI layperson here. Is there something like an MRI scan for a neural network?

Imagining I could take a foundation model, run it on my specialized task, and measure which regions of the neural network light up.

Then I could carve out unused regions of the network to create a more lightweight model.

Or is this a silly idea?

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#86

Post by jacquesm »

DesiLurker wrote: post #84
jacquesm wrote: post #37

Earlier quoted context omitted.

The rest of that difference can easily be explained by the difference in the class of hardware used. A desktop made today vs a laptop is roughly that factor 4. Not sure if back then there would have been laptops that you could have done this on for a more apples-to-apples comparison. Modern laptops give great efficiency, when I went for solar power here the first thing to go was the desktop computer. I still have it,…

also the 'memory wall', remember memory b/w did not grow at pace with moore's law. sure there are ways to mitigate it but that eat into chip budget and reflect when real world performance is calculated.

Yes, true and in a way that wall is still there. The way GPUs are limited in how much RAM they have because there is a way to sell you that memory at a multiple of the cost.

Imagine a GPU with a 128G or even 256G slot based memory section that is sold unpopulated. 8 SODIMM slots or so.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#87

Post by eigenvalue »

version_five wrote: post #2

This was really good. The only thing I didn't see explicitly discussed, although I guess it's obvious, is that what's different 33 years later is the inputs the models operate on. The '89 sota model used 16x16 greyscale images, today we have single digit megapixel color images, in 30 years, a desktop will be able to train Clip in 90 seconds, but what will the sota models be trained on?

We might have megapixel images that we can easily get with phone cameras, but virtually all vision models in common use take 224x224 resolution images as input, or maybe 384x384. Anything higher resolution than that just gets resampled down. It seems that you are better off using your compute budget on a bigger “brain” than on better “eyes” for now.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#88

Post by beebmam »

nojvek wrote: post #56
eru wrote: post #40

Earlier quoted context omitted.

It totally depends on what you want to use a measure for. Just like neither height or volume alone will tell you what will fit in your car. By any measure that puts energy used by the brain in the denominator, humans are probably dumber than ants. But that doesn't mean those measures are always accurate. (For contemporary neural networks, you also have to distinguish training costs from inference costs.)

To add more context, humans are 100W biological machines. Brain is ~20% of that power - 20W. The greatest form of general intelligence at 20W. A MacBook Air is ~30W. https://www.jackery.com/blogs/knowledge/how-many-watts-a-lap...

You're leaving out the training requirements

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#89

Post by version_five »

eigenvalue wrote: post #87
version_five wrote: post #2

This was really good. The only thing I didn't see explicitly discussed, although I guess it's obvious, is that what's different 33 years later is the inputs the models operate on. The '89 sota model used 16x16 greyscale images, today we have single digit megapixel color images, in 30 years, a desktop will be able to train Clip in 90 seconds, but what will the sota models be trained on?

We might have megapixel images that we can easily get with phone cameras, but virtually all vision models in common use take 224x224 resolution images as input, or maybe 384x384. Anything higher resolution than that just gets resampled down. It seems that you are better off using your compute budget on a bigger “brain” than on better “eyes” for now.

I don't think that's current. Certainly the object detection models work on bigger images, and the datasets they're pretrained on e.g. coco are not 224x224. I think standard models pretrained on imagenet, like the Resnets usually have everything resized to 224x224, and so they favor this kind of scaling.

Re: Deep Neural Nets: 33 years ago and 33 years from now (2022)

#90

Post by NumberWangMan »

dsign wrote: post #30

It's not clear that compute will scale as it did for the next 33 years. But it doesn't really need to. I read the article and I was thinking "my God, I remember I used MSE that weekend in my pet ML project and it really didn't work out that well; wrong loss function." Our current crop of LLMs, or the one next year, will be perfectly able to tell me how I can improve my code and graphs, which means that I can deploy s…

I think there’s going to be a point where we need to slow a AI way, way down in order to avoid bad outcomes. I’m with Zvi Mowshowitz here: we should encourage progress and risk taking in every area except those where there are extinction risks. Applying today’s LLMs to all sorts of problems won’t end us. But I think we may only be a few years away from AGI that is conscious and can plan, and we don’t know the upper limit of how smart we’ll be able to make them.

And I think that we have a responsibility to any intelligent being we bring into the world. Some lament that there’s no test to become a parent-what about creating a million copies of a new virtual brain from scratch? And basically so they can be born into lifelong servitude.

Post reply on HN