Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural Networks: Zero to Hero

karpathy.ai

Post reply on HN
Print view View original

71–80 of 88 posts

Re: Neural Networks: Zero to Hero

#71

Post by lazarus01 »

m3kw9 wrote: post #68

Does learning this still matter now?

Yes, the current technology cannot replace an engineer.

The easiest way to understand why is by understanding natural language. A natural language like english is very messy and and doesn't follow formal rules. It's also not specific enough to provide instructions to a computer, that's why code was created.

The AI is incredibly dumb when it comes to complex tasks with long range contexts. It needs an engineer that understands how to write and execute code to give it precise instructions or it is useless.

Natural Language Processing is so complex, it started around the end of world war two and we are just now seeing innovation in AI where we can mimmick humans, where the AI can do certain things faster than humans. But thinking is not one of them.

Re: Neural Networks: Zero to Hero

#72

Post by miki123211 »

ronbenton wrote: post #46
3abiton wrote: post #42

Earlier quoted context omitted.

Karpathy has a great intuitive style, but sometimes it's too dumbed down. If you come from adjacent fields, it might be a bit dragging, but it's always entertaining

>Karpathy has a great intuitive style, but sometimes it's too dumbed down As someone who has tried some teaching in the past, it's basically impossible to teach to an audience with a wide array of experience and knowledge. I think you need to define your intended audience as narrowly as possible, teach them, and just accept that more knowledgeable folk may be bored and less knowledgeable folk may be lost.

I think this is where LLM-assisted education is going to shine.

An LLM is the perfect tool to fill the little gaps that you need to fill to understand that one explanation that's almost at your level, but not quite.

Re: Neural Networks: Zero to Hero

#73

Post by mlmonkey »

ronbenton wrote: post #46
3abiton wrote: post #42

Earlier quoted context omitted.

Karpathy has a great intuitive style, but sometimes it's too dumbed down. If you come from adjacent fields, it might be a bit dragging, but it's always entertaining

>Karpathy has a great intuitive style, but sometimes it's too dumbed down As someone who has tried some teaching in the past, it's basically impossible to teach to an audience with a wide array of experience and knowledge. I think you need to define your intended audience as narrowly as possible, teach them, and just accept that more knowledgeable folk may be bored and less knowledgeable folk may be lost.

When I was an instructor for courses like "Intro to Programming", this was definitely the case. The students ranged from "have never programmed before" to "I've been writing games in my spare time", but because it was a prerequisite for other courses, they all had to do it.

Teaching the class was a pain in the ass! What seemed to work was to do the intro stuff, and periodically throw a bone to the smartasses. Once I had them on my side, it became smooth sailing.

Re: Neural Networks: Zero to Hero

#74

Post by ruraljuror »

HarHarVeryFunny wrote: post #67
p1esk wrote: post #65

Earlier quoted context omitted.

Strange question. If you don’t know why you need this, you probably don’t. It will be the same as with the introductory AI course you did 20 years ago.

Well, no ... For a start any "AI" course 20 years ago probably wouldn't have even mentioned neural nets, and certainly not as a mainstream technique. A 20yr old "AI" curriculum would have looked more like the 3rd edition of Russel & Norvig's "Artificial Intelligence - A Modern Approach". https://github.com/yanshengjia/ml-road/blob/master/resources... Karpathy's videos aren't an AI (except in modern sense of AI=LLMs)…

I think they meant the result— not the content—would be the same.

Re: Neural Networks: Zero to Hero

#75

Post by eps »

HarHarVeryFunny wrote: post #67
p1esk wrote: post #65

Earlier quoted context omitted.

Strange question. If you don’t know why you need this, you probably don’t. It will be the same as with the introductory AI course you did 20 years ago.

Well, no ... For a start any "AI" course 20 years ago probably wouldn't have even mentioned neural nets, and certainly not as a mainstream technique. A 20yr old "AI" curriculum would have looked more like the 3rd edition of Russel & Norvig's "Artificial Intelligence - A Modern Approach". https://github.com/yanshengjia/ml-road/blob/master/resources... Karpathy's videos aren't an AI (except in modern sense of AI=LLMs)…

Neural nets were taught in my Uni in the late 90s. They were presented as the AI technique, which was however computationally infeasible at the time. Moreover, it was clearly stated that all supporting ideas were developed and researched 20 years prior, and the field was basically stagnated due to hardware not being there.

Re: Neural Networks: Zero to Hero

#76

Post by CamperBob2 »

HarHarVeryFunny wrote: post #67
p1esk wrote: post #65

Earlier quoted context omitted.

Strange question. If you don’t know why you need this, you probably don’t. It will be the same as with the introductory AI course you did 20 years ago.

Well, no ... For a start any "AI" course 20 years ago probably wouldn't have even mentioned neural nets, and certainly not as a mainstream technique. A 20yr old "AI" curriculum would have looked more like the 3rd edition of Russel & Norvig's "Artificial Intelligence - A Modern Approach". https://github.com/yanshengjia/ml-road/blob/master/resources... Karpathy's videos aren't an AI (except in modern sense of AI=LLMs)…

Anyone who watches the videos and follows along will indeed come up to speed on the basics of neural nets, at least with respect to MLPs. It's an excellent introduction.

Re: Neural Networks: Zero to Hero

#77

Post by CamperBob2 »

lazarus01 wrote: post #71
m3kw9 wrote: post #68

Does learning this still matter now?

Yes, the current technology cannot replace an engineer. The easiest way to understand why is by understanding natural language. A natural language like english is very messy and and doesn't follow formal rules. It's also not specific enough to provide instructions to a computer, that's why code was created. The AI is incredibly dumb when it comes to complex tasks with long range contexts. It needs an engineer that un…

LOL. Figuring out how to solve IMO-level math problems without "thinking" would be even more impressive than thinking itself. Now there's a parrot I'd buy.

Re: Neural Networks: Zero to Hero

#78

Post by zingar »

HarHarVeryFunny wrote: post #67
p1esk wrote: post #65

Earlier quoted context omitted.

Strange question. If you don’t know why you need this, you probably don’t. It will be the same as with the introductory AI course you did 20 years ago.

Well, no ... For a start any "AI" course 20 years ago probably wouldn't have even mentioned neural nets, and certainly not as a mainstream technique. A 20yr old "AI" curriculum would have looked more like the 3rd edition of Russel & Norvig's "Artificial Intelligence - A Modern Approach". https://github.com/yanshengjia/ml-road/blob/master/resources... Karpathy's videos aren't an AI (except in modern sense of AI=LLMs)…

My introductory course used that exact textbook and I still have it on my shelf :).

It has a chapter or two on NNs and even mentions back propagation in the index, but the majority of the book focuses elsewhere.

Re: Neural Networks: Zero to Hero

#79

Post by lazarus01 »

CamperBob2 wrote: post #77
lazarus01 wrote: post #71

Earlier quoted context omitted.

Yes, the current technology cannot replace an engineer. The easiest way to understand why is by understanding natural language. A natural language like english is very messy and and doesn't follow formal rules. It's also not specific enough to provide instructions to a computer, that's why code was created. The AI is incredibly dumb when it comes to complex tasks with long range contexts. It needs an engineer that un…

LOL. Figuring out how to solve IMO-level math problems without "thinking" would be even more impressive than thinking itself. Now there's a parrot I'd buy.

It isn't thinking it's RL with reward hacking.

It's like taking a student who wins a gold in IMO math, but can't solve easier math problems, because they did not study those type of problems. Where a human who is good at IMO math generalizes to all math problems.

It's just memorizing a trajectory as part of a specific goal. That's what RL is.

Re: Neural Networks: Zero to Hero

#80

Post by HarHarVeryFunny »

CamperBob2 wrote: post #76
HarHarVeryFunny wrote: post #67

Earlier quoted context omitted.

Well, no ... For a start any "AI" course 20 years ago probably wouldn't have even mentioned neural nets, and certainly not as a mainstream technique. A 20yr old "AI" curriculum would have looked more like the 3rd edition of Russel & Norvig's "Artificial Intelligence - A Modern Approach". https://github.com/yanshengjia/ml-road/blob/master/resources... Karpathy's videos aren't an AI (except in modern sense of AI=LLMs)…

Anyone who watches the videos and follows along will indeed come up to speed on the basics of neural nets, at least with respect to MLPs. It's an excellent introduction.

Sure, the basics of neural nets, but it seems just as a foundation leading to LLMs. He doesn't cover the zoo of ANN architectures such as ResNets, RNNs, LSTMs, GANs, diffusion models, etc, and barely touches on regularization, optimization, etc, other than mentioning BatchNorm and promising ADAM in a later video.

It's a useful series of videos no doubt, but his goal is to strip things down to basics and show how an ANN like a Transformer can be built from the ground up without using all the tools/libraries that would actually be used in practice.

Post reply on HN