Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Implementing a Neural Network from Scratch in Python

victorzhou.com

Post reply on HN
Print view View original

91–100 of 104 posts

Re: Implementing a Neural Network from Scratch in Python

#91

Post by FridgeSeal »

YeGoblynQueenne wrote: post #89
jacobolus wrote: post #84

Earlier quoted context omitted.

> just because software does "math" doesn't change how people read it. It absolutely does. Different problem domains (and different communities’ treatment of problems) involve differing types and amounts of formal structure, differing conventional notations, etc., and in practice the code looks substantially different (in organization, abstractions used, naming, ...) even if you try to standardize it to all look the…

>> People who are reading “math” code can be expected to understand mathematical notation, e.g. to be capable of reading a journal paper where an algorithm or formula is described more completely including motivation, formal derivation, proofs of correctness, proofs of various formal properties, ... The majority of machine learning papers are very well stocked in terms of heavy mathematical-y notation, but are very,…

If you're working in a domain, is it really that much to ask to become familiar with it? Especially if the domain has a large theoretical component.

When we teach people software engineering we teach them concepts like "give your variables meaningful names". Now that we're in sub-domain of implementing some mathematics in software, I'd argue that matching the variables and functions to their source (more or less) _is_ exactly "giving your variables meaningful names".

> A great deal of complexity could be removed from machine learning papers by notating algorithms as algorithms rather than formulae

And you would immediately lose the ability to quickly and easily recognise similar patterns and abstractions that mathematical notation so fluently allows.

Re: Implementing a Neural Network from Scratch in Python

#92

Post by throwaway287391 »

gambler wrote: post #67
jacobolus wrote: post #60

Earlier quoted context omitted.

And when trying to do algebraic manipulation on paper, having an 8 or 10 letter variable name is incredibly cumbersome. Frankly in the middle of a numerical algorithm it is typically also cumbersome in code to have descriptive variable names for everything. However, mathematical code (especially when written by scientists, etc.) often takes this too far, introducing many 1- or few-letter variable names without enough…

>h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear ∀ abbr + to the cogntv load prsn wht mntn in thr hd. ∀ -> every + -> adds wht -> who has to h -> head Was that easier to read than the following? Every abbreviation adds to the cognitive load the person has to maintain in their head. This isn't code, but the principles of making code (un)readable are exactly the same. If you see that full-wo…

"set_to_random_numbers" is much less descriptive than np.random.normal...the former doesn't tell me what distribution the random numbers come from. Agreed that the weights should be stored in a list or the like in real life rather than duplicating code for each layer, but the examples in the blog post are clearly intentionally minimizing abstractions as much as possible so that an untrained reader can immediately tell exactly what each line is doing. It's obviously not intended to scale to larger models.

Re: Implementing a Neural Network from Scratch in Python

#93

Post by cgopalan »

I took the Pytorch scholarship challenge this past December with Udacity. I couldn't make it to the nanodegree scholarship but the challenge course was really good in exposing me to the various facets of deep learning. The notebooks are free, but the videos really help explain some concepts well. For people who took the course, the videos are available for another year, so that helps.

I wanted to get a good grasp of DL but was floundering around before with so many sources and tutorials - and frameworks like Keras, Pytorch, TensorFlow etc. This course helped me make some decisions - like if I needed to get a developer's handle on DL, I would have to know Perceptrons, CNNs, RNNs, Style Transfer, Sentiment Prediction and GANs (even though GANs were not part of the course). And Pytorch would be my tool of choice.

Re: Implementing a Neural Network from Scratch in Python

#95

Post by theblackcat1002 »

I think a better and elegant way to implement is the pytorch way. By defining a forward and backward function for easy chaining, you can easily chain multiple layers into a single model. Placing the training loop inside the layer will be hard to write a multi layer version.

Just wrote a similar from scratch code few months back: https://github.com/theblackcat102/DL_2018/blob/master/HW2/la...

Re: Implementing a Neural Network from Scratch in Python

#96

Post by nilkn »

gambler wrote: post #32

Why are deep learning researchers allergic to meaningful variable names?

For what it's worth, I didn't read the article at all, went straight to the final code, and had no issue reading or understanding it. The variable names seem fine to me. They're actually quite good, and I liked the convention for naming partial derivatives.

This code is designed to be read by someone who is pretty familiar with the underlying concepts. If you don't really know how a neural network works or how gradient descent is applied to update the weights, that's your blocker, not the variable names I saw here.

Re: Implementing a Neural Network from Scratch in Python

#97

Post by deytempo »

nilkn wrote: post #96
gambler wrote: post #32

Why are deep learning researchers allergic to meaningful variable names?

For what it's worth, I didn't read the article at all, went straight to the final code, and had no issue reading or understanding it. The variable names seem fine to me. They're actually quite good, and I liked the convention for naming partial derivatives. This code is designed to be read by someone who is pretty familiar with the underlying concepts. If you don't really know how a neural network works or how gradie…

Well then they should get rid of the misleading title “from scratch” it makes it sound like they are going to explain it without beltching out a bunch of abstract math. I’m really annoyed that everyone pretends that you need math to explain a NN. I have done plenty of them without numpy or math using just arrays and loops.

Re: Implementing a Neural Network from Scratch in Python

#98

Post by deytempo »

ddebernardy wrote: post #54

One of the nice parts of NNs is that it takes some maths education to understand what's going on. My initial thought when reading the title was: "uh oh, are AI tutorials going to be the next PHP/MySQL tutorials, with god awful code all over the internet?" They haven't yet insofar as I'm aware. I hope the maths involved will prevent them from becoming that.

Luckily math isn’t actually required to implement them in a series of arrays so that little relief will be short lived.

Re: Implementing a Neural Network from Scratch in Python

#99

Post by zaphirplane »

FridgeSeal wrote: post #91
YeGoblynQueenne wrote: post #89

Earlier quoted context omitted.

>> People who are reading “math” code can be expected to understand mathematical notation, e.g. to be capable of reading a journal paper where an algorithm or formula is described more completely including motivation, formal derivation, proofs of correctness, proofs of various formal properties, ... The majority of machine learning papers are very well stocked in terms of heavy mathematical-y notation, but are very,…

If you're working in a domain, is it really that much to ask to become familiar with it? Especially if the domain has a large theoretical component. When we teach people software engineering we teach them concepts like "give your variables meaningful names". Now that we're in sub-domain of implementing some mathematics in software, I'd argue that matching the variables and functions to their source (more or less) _is…

You have hit the nail on the head meaningful names != long names

Re: Implementing a Neural Network from Scratch in Python

#100

Post by dragandj »

victor106 wrote: post #72
dragandj wrote: post #18

Earlier quoted context omitted.

I currently publish a series of detailed articles that cover exactly that. https://dragan.rocks/articles/19/Deep-Learning-in-Clojure-Fr...

dragon.rocks, rocks. Your articles are some of the best i have seen on this topic. Thank you. I hope you write a book on this topic.

Thanks! We'll see. BTW, its dragAn, not dragOn :)
Post reply on HN