Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Implementing a Neural Network from Scratch in Python

victorzhou.com

Post reply on HN
Print view View original

81–90 of 104 posts

Re: Implementing a Neural Network from Scratch in Python

#81

Post by gambler »

jacobolus wrote: post #71
gambler wrote: post #67

Earlier quoted context omitted.

>h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear ∀ abbr + to the cogntv load prsn wht mntn in thr hd. ∀ -> every + -> adds wht -> who has to h -> head Was that easier to read than the following? Every abbreviation adds to the cognitive load the person has to maintain in their head. This isn't code, but the principles of making code (un)readable are exactly the same. If you see that full-wo…

Are you suggesting that we should write all of our mathematical expressions using prose, and stop using symbols for operators? That would be the original approach historically (before the past 500 years), e.g. for the quadratic formula, Brahmagupta (628 CE): > To the absolute number multiplied by four times the square, add the square of the middle term; the square root of the same, less the middle term, being divided…

I am suggesting that just because software does "math" doesn't change how people read it. Bad coding practices like cryptic variable names and repetitions will affect it exactly in the same ways they affect software in all other domains.

Re: Implementing a Neural Network from Scratch in Python

#82

Post by applecrazy »

syntaxing wrote: post #49

If you're interested in learning something like this, I highly recommend Udacity's deep learning nanodegree. They have a section that teaches you how to build your own neural network with the the help of numpy. They even have a section where you write your own sentimental analysis neural network from scratch. Easily the best explanation and notebook to follow I have seen so far. I've been taking their revamped one th…

The notebooks are available on GitHub for free.

Re: Implementing a Neural Network from Scratch in Python

#83

Post by ska »

gambler wrote: post #81
jacobolus wrote: post #71

Earlier quoted context omitted.

Are you suggesting that we should write all of our mathematical expressions using prose, and stop using symbols for operators? That would be the original approach historically (before the past 500 years), e.g. for the quadratic formula, Brahmagupta (628 CE): > To the absolute number multiplied by four times the square, add the square of the middle term; the square root of the same, less the middle term, being divided…

I am suggesting that just because software does "math" doesn't change how people read it. Bad coding practices like cryptic variable names and repetitions will affect it exactly in the same ways they affect software in all other domains.

I see what you are saying, but context matters.

In code that starts something like "/* implements gambler et. al 2019 (doi:xxxxxxx) eqn 3 ... */". Then I really expect the code to go to great lengths to match the notation used in the paper. Anything else is adding to the cognitive load.

The exception is if the entire algorithm is discussed in the comments of the code without outside reference, then I want the code and comment to be extremely consistent.

Personally, I like the former as a shorthand for "don't mess with this without the paper in front of you, you'll probably screw it up".

Re: Implementing a Neural Network from Scratch in Python

#84

Post by jacobolus »

gambler wrote: post #81
jacobolus wrote: post #71

Earlier quoted context omitted.

Are you suggesting that we should write all of our mathematical expressions using prose, and stop using symbols for operators? That would be the original approach historically (before the past 500 years), e.g. for the quadratic formula, Brahmagupta (628 CE): > To the absolute number multiplied by four times the square, add the square of the middle term; the square root of the same, less the middle term, being divided…

I am suggesting that just because software does "math" doesn't change how people read it. Bad coding practices like cryptic variable names and repetitions will affect it exactly in the same ways they affect software in all other domains.

> just because software does "math" doesn't change how people read it.

It absolutely does. Different problem domains (and different communities’ treatment of problems) involve differing types and amounts of formal structure, differing conventional notations, etc., and in practice the code looks substantially different (in organization, abstractions used, naming, ...) even if you try to standardize it to all look the same.

People who are reading “math” code can be expected to understand mathematical notation, e.g. to be capable of reading a journal paper where an algorithm or formula is described more completely including motivation, formal derivation, proofs of correctness, proofs of various formal properties, ...

Mathematical code is often quite abstract; unlike company-specific business logic, the same tool might be used in many different contexts with inputs of different meanings. There really isn’t that much insight gained by replacing i with “index”, x with “generic_input_variable”,

or to take it to an extreme, ax + b with add(multiply(input_variable, proportionality_constant), offset_constant)

or sin(x) with perpendicular_unit_vector_component_for_angle(input_angle_measure)

The extra space overhead of the long variables and words instead of symbols is a killer for clarity.

If variable names are “cryptic” [as in, can’t be guessed at a glance by someone working in the field] then that is indeed a failure though. Short variable names should have limited scope (ideally fitting on one screenfull of code) and obvious meaning in context, which might involve some explanatory comments, links to a paper.

Re: Implementing a Neural Network from Scratch in Python

#85

Post by siekmanj »

fartcannon wrote: post #68
siekmanj wrote: post #41

Earlier quoted context omitted.

shameless self-promotion: https://github.com/siekmanj/sieknet

Looks great. Even got yourself a genetic algorithm in there, eh? How well does it work?

The genetic algorithm currently isn't in a working state unfortunately. I did a big architectural rewrite recently and haven't gotten around to GA stuff - but I'm looking forward to applying some more modern results in neuroevolution from the last two or three years.

Re: Implementing a Neural Network from Scratch in Python

#86

Post by avmich »

cr0sh wrote: post #66
sgillen wrote: post #26

Earlier quoted context omitted.

I still think it's a useful exercise for people like me who learn best by implementing a thing. Tensorflow, Pytorch, etc introduce a lot of magic for someone who is completely unfamiliar with the area.

If you want a great introduction, check out Andrew Ng's Coursera course: https://www.coursera.org/learn/machine-learning This is the same course that was originally called "ML Class" in 2011 when it was offered via Stanford (it, plus the other course called "AI Class" - helped to launch Coursera and Udacity, respectively). It uses Octave as it's development language to teach the concepts. Octave has as it's "primitiv…

This is a great course, I took it once a few years ago and sometime think if I should take it again. Wonder how it changed over time. It was eye-opening training a small NN by hand on MNIST dataset and getting it to recognize digits surprisingly well.

However even understanding all that doesn't allow one using modern frameworks by itself, they seem to carry too much additional assumptions, terminology which aren't quite explained with similar level of patience.

Re: Implementing a Neural Network from Scratch in Python

#88

Post by YeGoblynQueenne »

gambler wrote: post #67
jacobolus wrote: post #60

Earlier quoted context omitted.

And when trying to do algebraic manipulation on paper, having an 8 or 10 letter variable name is incredibly cumbersome. Frankly in the middle of a numerical algorithm it is typically also cumbersome in code to have descriptive variable names for everything. However, mathematical code (especially when written by scientists, etc.) often takes this too far, introducing many 1- or few-letter variable names without enough…

>h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear ∀ abbr + to the cogntv load prsn wht mntn in thr hd. ∀ -> every + -> adds wht -> who has to h -> head Was that easier to read than the following? Every abbreviation adds to the cognitive load the person has to maintain in their head. This isn't code, but the principles of making code (un)readable are exactly the same. If you see that full-wo…

In Prolog it's very common to use single- or two-letter names for variables (which are all upper-case). The following is a typical example:

  member(X,[X|_Xs]).
  member(X,[_|Xs]):-
         member(X,Xs).
("|" is the cons operator in Prolog. An underscore means a don't-care variable).

In cases like this I think a "meaningful" variable name would actively hamper reading. For example:

  member(Element,[Element|_Rest_of_list]).
  member(Element,[_|Rest_of_list]) :-
         member(Element,Rest_of_list).
Personally, I stick to using a capital letter followed by an "s" for a list and the same capital letter alone for an element of a list, as above, though that is by no means a universal convention.

It's also common to use variables I,J,K,N,M for numbers and P,Q,R for predicate symbols (that can be passed as arguments to predicates, in Prolog). If the code sticks to the same convention throughout, it gets much easier to read than having to come up with special names for each variable ("Index", "Counter", "Next_value", "Length", etc).

And, if I remember correctly, the same kind of convention is common in Haskell, where I understand you can actualy "dash" variables (as in x, x').

Re: Implementing a Neural Network from Scratch in Python

#89

Post by YeGoblynQueenne »

jacobolus wrote: post #84
gambler wrote: post #81

Earlier quoted context omitted.

I am suggesting that just because software does "math" doesn't change how people read it. Bad coding practices like cryptic variable names and repetitions will affect it exactly in the same ways they affect software in all other domains.

> just because software does "math" doesn't change how people read it. It absolutely does. Different problem domains (and different communities’ treatment of problems) involve differing types and amounts of formal structure, differing conventional notations, etc., and in practice the code looks substantially different (in organization, abstractions used, naming, ...) even if you try to standardize it to all look the…

>> People who are reading “math” code can be expected to understand mathematical notation, e.g. to be capable of reading a journal paper where an algorithm or formula is described more completely including motivation, formal derivation, proofs of correctness, proofs of various formal properties, ...

The majority of machine learning papers are very well stocked in terms of heavy mathematical-y notation, but are very, very low on formal derivation, proofs of correctness, proofs of anything like formal properties, or even motivation ("wait, where did this vector come from?"). Most have no theoretical results at all- only definitions.

So let's not overdo it. The OP is making a reasonable demand: write complex code in a way that makes it easily readable without being part of an elite brotherood of adepts that know all the secret handshakes and shibboleths.

A great deal of complexity could be removed from machine learning papers by notating algorithms as algorithms rather than formulae. For example, you can say exactly the same thing with two "for i to j" and two summations with top and bottom indices. Sometimes the mathematical notation can be more compact- but when your subscripts start having subscripted superscripts, it's time to stop and think what you're trying to do.

Besides- the OP did talk about code not papers. Code has to be maintained by someone, usually someone else. Papers, not so much.

Re: Implementing a Neural Network from Scratch in Python

#90

Post by FridgeSeal »

gambler wrote: post #67
jacobolus wrote: post #60

Earlier quoted context omitted.

And when trying to do algebraic manipulation on paper, having an 8 or 10 letter variable name is incredibly cumbersome. Frankly in the middle of a numerical algorithm it is typically also cumbersome in code to have descriptive variable names for everything. However, mathematical code (especially when written by scientists, etc.) often takes this too far, introducing many 1- or few-letter variable names without enough…

>h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear ∀ abbr + to the cogntv load prsn wht mntn in thr hd. ∀ -> every + -> adds wht -> who has to h -> head Was that easier to read than the following? Every abbreviation adds to the cognitive load the person has to maintain in their head. This isn't code, but the principles of making code (un)readable are exactly the same. If you see that full-wo…

In the context of the equation, having long/non-matching variable names ads to the overhead. I find it orders of magnitude easier to read some Julia code examples that use matching symbols than I do the equivalent thing in python with full variable names.

If I'm implementing some complex equation I try to match the symbols as much as possible and keep the representation compact because it means when I (or my teammates) come back to the code they can see the whole complex thing in one go and easily recognise the equation it comes from.

Post reply on HN