Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Implementing a Neural Network from Scratch in Python

victorzhou.com

Post reply on HN
Print view View original

21–30 of 104 posts

Re: Implementing a Neural Network from Scratch in Python

#21

Post by fit2rule »

dbcurtis wrote: post #15
fredley wrote: post #2

My first reading of this headline was "Implementing a Neural Network in Scratch". Now that I would like to see. https://scratch.mit.edu/

That would be a challenge. Back in the heyday of expert systems and original Mac’s, I did an expert system in Hypercard. Not really production worthy. But interesting from perspective of exploring UI.

Still got it somewhere? I've got a project to demonstrate a bunch of Hypercard stacks in a retro-computing context, and it'd be definitely interesting to see, in the case that you'd consider finding it and sharing it ...

Re: Implementing a Neural Network from Scratch in Python

#22

Post by jventura »

eam wrote: post #20

I didn't have time to read all of it, but half way through. So far it has bee a pretty good write up, however, just one question. Can the author or anyone explain how were the amount for values shifted chosen? I.e. Weight (minus 135) and Height (minus 66). Why was -135 and -66 chosen? An explanation would be helpful. Thanks!

I think they were chosen such that the weight and height values become negative for Females and positive for Males.

Re: Implementing a Neural Network from Scratch in Python

#23

Post by CodeMage »

11thEarlOfMar wrote: post #10

"We’ll use an optimization algorithm called stochastic gradient descent (SGD) that tells us how to change our weights and biases to minimize loss. It’s basically just this update equation: w1 ← w1 − η ∂w|∂L η is a constant called the learning rate that controls how fast we train. All we’re doing is subtracting η ∂w|∂L from w1: - If ∂L | ∂w1 is positive, w1 will decrease, which makes L decrease - If ∂L | ∂w1 is positi…

There's are a couple of misquotes there. For starters, it's not "w1 ← w1 − η ∂w|∂L" but "w1 ← w1 − η ∂L|∂w1". Also, the second "if" should say "negative", not "positive" again.

In case you were being sarcastic, it is simple if you don't take it out of context. It requires a certain understanding of derivatives, but a reader who doesn't understand them won't make it to the point you're quoting.

Re: Implementing a Neural Network from Scratch in Python

#24

Post by yorwba »

muttled wrote: post #9
inputcoffee wrote: post #3

There are so many little details to remember when you implement a Neural Network from "scratch". Or, I suppose, even if you do not. You know what would be a great contribution? An extensive set of unit tests, or even just problems with solutions. That way people can write their own implementations and test them. And even if a person were to implement the net in Pytorch of Tensorflow, they could test the work. So ther…

Built-in testing would be fantastic. Being able to tell if you designed a model wrong or just made an error setting up the code would reduce frustration a ton!

If you're implementing your own derivatives, you should probably use numerical methods to check that the gradient is computed correctly, and pytorch comes with tools to help with that https://pytorch.org/docs/stable/autograd.html#numerical-grad...

If you're reimplementing something for learning purposes, you can just compare the output you get with the existing implementation.

But as soon as you're trying to do something novel, no automatic test can tell you whether the model architecture or your implementation of it is at fault when you don't get the results you'd like, because there's nothing you could use for comparison.

Re: Implementing a Neural Network from Scratch in Python

#25

Post by avmich »

> Real neural net code looks nothing like this.

This negates a lot of promise of the article. If one understands principles, but needs to know how to apply them, this statement makes the article practically useless.

Usually after reading like this goes a jump in explanations, which directs to apply TensorFlow et. al. and even, if you're lucky, explains how to do that - but doesn't explain what it is that TensorFlow does qualitatively different, which would justify putting such a phrase as a warning.

Re: Implementing a Neural Network from Scratch in Python

#26

Post by sgillen »

avmich wrote: post #25

> Real neural net code looks nothing like this. This negates a lot of promise of the article. If one understands principles, but needs to know how to apply them, this statement makes the article practically useless. Usually after reading like this goes a jump in explanations, which directs to apply TensorFlow et. al. and even, if you're lucky, explains how to do that - but doesn't explain what it is that TensorFlow d…

I still think it's a useful exercise for people like me who learn best by implementing a thing.

Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area.

Re: Implementing a Neural Network from Scratch in Python

#27

Post by xeRTRex »

eam wrote: post #20

I didn't have time to read all of it, but half way through. So far it has bee a pretty good write up, however, just one question. Can the author or anyone explain how were the amount for values shifted chosen? I.e. Weight (minus 135) and Height (minus 66). Why was -135 and -66 chosen? An explanation would be helpful. Thanks!

I haven't read the article thoroughly but I suspect part of the data preprocessing step included centering and standardizing. Those must be the mean weight and height values.

Re: Implementing a Neural Network from Scratch in Python

#28

Post by LodeOfCode »

eam wrote: post #20

I didn't have time to read all of it, but half way through. So far it has bee a pretty good write up, however, just one question. Can the author or anyone explain how were the amount for values shifted chosen? I.e. Weight (minus 135) and Height (minus 66). Why was -135 and -66 chosen? An explanation would be helpful. Thanks!

"I arbitrarily chose the shift amounts (135 and 66) to make the numbers look nice. Normally, you’d shift by the mean."

Re: Implementing a Neural Network from Scratch in Python

#29

Post by dbcurtis »

fit2rule wrote: post #21
dbcurtis wrote: post #15

Earlier quoted context omitted.

That would be a challenge. Back in the heyday of expert systems and original Mac’s, I did an expert system in Hypercard. Not really production worthy. But interesting from perspective of exploring UI.

Still got it somewhere? I've got a project to demonstrate a bunch of Hypercard stacks in a retro-computing context, and it'd be definitely interesting to see, in the case that you'd consider finding it and sharing it ...

Oh gosh.... that was so many years ago. I suspect I could never find the floppy, assuming I haven't chucked it.

All I can remember about it is that it was structured to ask the user one query per card, and there was some kind of global function that kept the state of the consultation and navigated to the next card.

The expert system that I implemented was a simple toy, because the point was to explore the UI aspects. For instance, you could easily embed images and ask "Click the image that most closely resembles your specimen." Stuff like that, intermixed with traditional text queries.

Re: Implementing a Neural Network from Scratch in Python

#30

Post by avmich »

sgillen wrote: post #26
avmich wrote: post #25

> Real neural net code looks nothing like this. This negates a lot of promise of the article. If one understands principles, but needs to know how to apply them, this statement makes the article practically useless. Usually after reading like this goes a jump in explanations, which directs to apply TensorFlow et. al. and even, if you're lucky, explains how to do that - but doesn't explain what it is that TensorFlow d…

I still think it's a useful exercise for people like me who learn best by implementing a thing. Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area.

> I still think it's a useful exercise for people like me who learn best by implementing a thing.

Yes, but if you implement one thing, and then TensorFlow which you plan to use does something very different, what's the point? Ok, different enough to justify the warning.

> Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area.

Exactly. It's that magic which is interesting, after you're shown how in principle - but only in principle - the thing could work.

Post reply on HN