Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Implementing a Neural Network from Scratch in Python

victorzhou.com

Post reply on HN
Print view View original

71–80 of 104 posts

Re: Implementing a Neural Network from Scratch in Python

#71

Post by jacobolus »

gambler wrote: post #67
jacobolus wrote: post #60

Earlier quoted context omitted.

And when trying to do algebraic manipulation on paper, having an 8 or 10 letter variable name is incredibly cumbersome. Frankly in the middle of a numerical algorithm it is typically also cumbersome in code to have descriptive variable names for everything. However, mathematical code (especially when written by scientists, etc.) often takes this too far, introducing many 1- or few-letter variable names without enough…

>h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear ∀ abbr + to the cogntv load prsn wht mntn in thr hd. ∀ -> every + -> adds wht -> who has to h -> head Was that easier to read than the following? Every abbreviation adds to the cognitive load the person has to maintain in their head. This isn't code, but the principles of making code (un)readable are exactly the same. If you see that full-wo…

Are you suggesting that we should write all of our mathematical expressions using prose, and stop using symbols for operators?

That would be the original approach historically (before the past 500 years), e.g. for the quadratic formula, Brahmagupta (628 CE):

> To the absolute number multiplied by four times the square, add the square of the middle term; the square root of the same, less the middle term, being divided by twice the square is the value.

Go ahead and do what you like, but I doubt you’ll find many publishers who will accept your paper in the 21st century.

I know which one imposes more cognitive load for me. But disclaimer: I spent a lot of time from age 5–20 working with mathematical notation.

Re: Implementing a Neural Network from Scratch in Python

#72

Post by victor106 »

dragandj wrote: post #18
kowdermeister wrote: post #14

I started playing around with NN-s recently and this looked like a good tutorial until the partial derivatives :) It would have been so much better to turn a complex math concept like that to code in smaller chunks because it's hard to extract from the full sample which part refers to which equation. Also, the bias is not explained at all. - Why is it there? - What does it do? - What value should it have?

I currently publish a series of detailed articles that cover exactly that. https://dragan.rocks/articles/19/Deep-Learning-in-Clojure-Fr...

dragon.rocks, rocks. Your articles are some of the best i have seen on this topic. Thank you. I hope you write a book on this topic.

Re: Implementing a Neural Network from Scratch in Python

#73

Post by Shorel »

avmich wrote: post #25

> Real neural net code looks nothing like this. This negates a lot of promise of the article. If one understands principles, but needs to know how to apply them, this statement makes the article practically useless. Usually after reading like this goes a jump in explanations, which directs to apply TensorFlow et. al. and even, if you're lucky, explains how to do that - but doesn't explain what it is that TensorFlow d…

> If one understands principles

In that case, the hard part is already done.

Understanding the principles is what takes the most work, not writing or adapting the code.

In fact, understanding the principles is IMO a prerequisite for being able to successfully adapt the code.

Now, using something like TensorFlow in another later article is of course super useful. It also will clarify why TensorFlow is more confusing and why this article was written.

I find the article enlightening, and specific libraries like TensorFlow are just "implementation details".

Re: Implementing a Neural Network from Scratch in Python

#74

Post by Shorel »

jackallis wrote: post #6

how come every information like this, intended for beginners, don't say why you even need NN or when would you even use it.

Why do you need to know linear algebra or calculus in programming?

A poignant answer is: you never really know! The future will surprise you!

Watching the movie about AlphaGo and all other stuff DeepMind has done will give you all the answers you need about the why NN.

The question in the first line has an obvious answer and it is just rhetorical. The bigger point is: this mindset doesn't apply just to Neural Networks, many things in mathematics are useful years after they are first invented, and it is better to learn general principles and have an open mind, Richard Feynman style.

Re: Implementing a Neural Network from Scratch in Python

#77

Post by Shorel »

rrggrr wrote: post #5

Great, but the problem with these summaries is no real-world data/use-case. An actual business case example with code is easier to comprehend for less math-inclined folks.

With neural networks, real-world data and real-world use-cases have expensive hardware requirements.

You can, however, try this service for free during the first two months: https://aws.amazon.com/blogs/aws/sagemaker/

(I am not affiliated with AWS and I have not used SageMaker)

You can also read the DeepMind blog to find out the latest state-of-the-art real-world usage of Neural Networks.

Re: Implementing a Neural Network from Scratch in Python

#78

Post by [deleted] »

jacobolus wrote: post #71
gambler wrote: post #67

Earlier quoted context omitted.

>h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear ∀ abbr + to the cogntv load prsn wht mntn in thr hd. ∀ -> every + -> adds wht -> who has to h -> head Was that easier to read than the following? Every abbreviation adds to the cognitive load the person has to maintain in their head. This isn't code, but the principles of making code (un)readable are exactly the same. If you see that full-wo…

Are you suggesting that we should write all of our mathematical expressions using prose, and stop using symbols for operators? That would be the original approach historically (before the past 500 years), e.g. for the quadratic formula, Brahmagupta (628 CE): > To the absolute number multiplied by four times the square, add the square of the middle term; the square root of the same, less the middle term, being divided…

[deleted]

Re: Implementing a Neural Network from Scratch in Python

#79

Post by chrisfosterelli »

gambler wrote: post #67
jacobolus wrote: post #60

Earlier quoted context omitted.

And when trying to do algebraic manipulation on paper, having an 8 or 10 letter variable name is incredibly cumbersome. Frankly in the middle of a numerical algorithm it is typically also cumbersome in code to have descriptive variable names for everything. However, mathematical code (especially when written by scientists, etc.) often takes this too far, introducing many 1- or few-letter variable names without enough…

>h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear ∀ abbr + to the cogntv load prsn wht mntn in thr hd. ∀ -> every + -> adds wht -> who has to h -> head Was that easier to read than the following? Every abbreviation adds to the cognitive load the person has to maintain in their head. This isn't code, but the principles of making code (un)readable are exactly the same. If you see that full-wo…

> Every abbreviation adds to the cognitive load the person has to maintain in their head.

I think your claim that abbreviations are always more cognitive load is wrong. For the sake of argument, let's take it at face value though. These are not even code abbreviations. They are literal translations to the math notation. The input "x" is not an abbreviation, it's a well defined value in neural network terminology.

If they called it something more descriptive like "input_vector_to_first_layer_of_neural_net", this would be more cognitive load because someone reviewing this now needs to mentally map this to "x" in the algorithm anytime they are reviewing both.

Now, to your claim itself, I think it's unfair to say abbreviations are always more cognitive load. These variables are significantly more approachable because they follow convention. I see "x" and I know what that is. If every individual developer went ahead and rewrote every neural net with something they viewed as more interpretable in their personal context, it'd be a lot harder to understand what is going on for everyone. The variable itself may be longer, but I might actually need a prose explanation of the variable's purpose because now I don't have the context of well established naming convention.

Re: Implementing a Neural Network from Scratch in Python

#80

Post by applecrazy »

kowdermeister wrote: post #14

I started playing around with NN-s recently and this looked like a good tutorial until the partial derivatives :) It would have been so much better to turn a complex math concept like that to code in smaller chunks because it's hard to extract from the full sample which part refers to which equation. Also, the bias is not explained at all. - Why is it there? - What does it do? - What value should it have?

Partial derivatives are basically how much a multivariate function changes based on a tiny change in a single variable. The actual math is irrelevant.
Post reply on HN