Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Implementing a Neural Network from Scratch in Python

victorzhou.com

Post reply on HN
Print view View original

31–40 of 104 posts

Re: Implementing a Neural Network from Scratch in Python

#31

Post by sgillen »

inputcoffee wrote: post #3

There are so many little details to remember when you implement a Neural Network from "scratch". Or, I suppose, even if you do not. You know what would be a great contribution? An extensive set of unit tests, or even just problems with solutions. That way people can write their own implementations and test them. And even if a person were to implement the net in Pytorch of Tensorflow, they could test the work. So ther…

That would be a great contribution but unfortunately I think not an easy one to provide. A few issues

1. Most BLAS libraries aren't deterministic. Floating point numbers aren't in general associative, (a + b) + c =/= a + (b + c). However most BLAS libraries will happily shuffle the order of operations around if it saves some CPU cycles, I've seen this issue cause huge differences in behavior between different CPUs before, although I think that was a pathological case. So if you want to do a test like this you really need to have everyone on the same page an installing a deterministic BLAS.

2. Training neural networks is stochastic I know you are talking about just the forward pass but I think it would be interesting to have these tests for training your network as well. Naively this sounds like it might be easy to fix (just provide a seed!), but gets very tricky when you want something to output the same code over multiple libraries, operating systems, hardware etc. Multithreading code, putting stuff on the GPU or even the cloud add even more difficulty.

Re: Implementing a Neural Network from Scratch in Python

#33

Post by thatoneuser »

gambler wrote: post #32

Why are deep learning researchers allergic to meaningful variable names?

They're typically not coders as a background. They're usually a hard science/math person so they didn't get variable indoctrination.

I haven't coded a NN since grad school and just scanning over the code I can tell this is basically the same as what I did. No doubt this is a boiler plate approach compiled into an article.

Re: Implementing a Neural Network from Scratch in Python

#34

Post by chrisfosterelli »

gambler wrote: post #32

Why are deep learning researchers allergic to meaningful variable names?

These are meaningful variable names in the context of a code implementation of a math algorithm. The variables directly map to the algorithmic notation.

Re: Implementing a Neural Network from Scratch in Python

#35

Post by sgillen »

avmich wrote: post #30
sgillen wrote: post #26

Earlier quoted context omitted.

I still think it's a useful exercise for people like me who learn best by implementing a thing. Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area.

> I still think it's a useful exercise for people like me who learn best by implementing a thing. Yes, but if you implement one thing, and then TensorFlow which you plan to use does something very different, what's the point? Ok, different enough to justify the warning. > Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area. Exactly. It's that magic which is interes…

I guess the point here is that in this article we implement some of the magic that the higher level frameworks use, which I think will be useful later when writing and debugging our high level code.

But I see your point which is that it would be nice if the article was telling us which part of the high level library we're implementing and what that would look like.

I'm still happy with the article though and would not call it almost useless. But hey if you'd rather jump right into a tensorkerastorch tutorial that's an option too!

Re: Implementing a Neural Network from Scratch in Python

#36

Post by xeRTRex »

kowdermeister wrote: post #14

I started playing around with NN-s recently and this looked like a good tutorial until the partial derivatives :) It would have been so much better to turn a complex math concept like that to code in smaller chunks because it's hard to extract from the full sample which part refers to which equation. Also, the bias is not explained at all. - Why is it there? - What does it do? - What value should it have?

Agreed, it is a little confusing. What makes it even more confusing is bias can be introduced as a regularization technique (to address overfitting) and these are not the same thing.

Bias in this context can be simply interpreted as analogous to the intercept in y = mx + b. Kind of like the baseline when no additional information is given.

Re: Implementing a Neural Network from Scratch in Python

#38

Post by avmich »

sgillen wrote: post #35
avmich wrote: post #30

Earlier quoted context omitted.

> I still think it's a useful exercise for people like me who learn best by implementing a thing. Yes, but if you implement one thing, and then TensorFlow which you plan to use does something very different, what's the point? Ok, different enough to justify the warning. > Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area. Exactly. It's that magic which is interes…

I guess the point here is that in this article we implement some of the magic that the higher level frameworks use, which I think will be useful later when writing and debugging our high level code. But I see your point which is that it would be nice if the article was telling us which part of the high level library we're implementing and what that would look like. I'm still happy with the article though and would no…

> I'm still happy with the article though and would not call it almost useless.

I really like the article. My problem is the next step, which I look for, rather unsuccessfully.

> But hey if you'd rather jump right into a tensorkerastorch tutorial that's an option too!

Do you know good tutorials for that? Preferably on the level of this article, only regarding those libraries?

Re: Implementing a Neural Network from Scratch in Python

#39

Post by eggie5 »

profunctor wrote: post #13
11thEarlOfMar wrote: post #10

"We’ll use an optimization algorithm called stochastic gradient descent (SGD) that tells us how to change our weights and biases to minimize loss. It’s basically just this update equation: w1 ← w1 − η ∂w|∂L η is a constant called the learning rate that controls how fast we train. All we’re doing is subtracting η ∂w|∂L from w1: - If ∂L | ∂w1 is positive, w1 will decrease, which makes L decrease - If ∂L | ∂w1 is positi…

I think it's supposed to be dL/dw1. As in the loss function derived wrt w1.

+1 partial of the loss w/ respect to the weight

Re: Implementing a Neural Network from Scratch in Python

#40

Post by sgillen »

avmich wrote: post #38
sgillen wrote: post #35

Earlier quoted context omitted.

I guess the point here is that in this article we implement some of the magic that the higher level frameworks use, which I think will be useful later when writing and debugging our high level code. But I see your point which is that it would be nice if the article was telling us which part of the high level library we're implementing and what that would look like. I'm still happy with the article though and would no…

> I'm still happy with the article though and would not call it almost useless. I really like the article. My problem is the next step, which I look for, rather unsuccessfully. > But hey if you'd rather jump right into a tensorkerastorch tutorial that's an option too! Do you know good tutorials for that? Preferably on the level of this article, only regarding those libraries?

Pytorch's official one is very good: https://pytorch.org/tutorials/beginner/pytorch_with_examples...

They even start with NumPy, as the article does.

Post reply on HN