Implementing a Neural Network from Scratch in Python

I am pretty sure that note wasn't written there before. The author must have updated the article after I posted this question. Though "nice" is still somewhat vague to me. Nice in what way? Smaller numbers are nicer? Negative and positives make it nice?

The second part of the answer is more useful information, "Normally, you’d shift by the mean."

I think this a form of normalizing a normal distribution into a standard normal distribution ( a normal distribution has two parameters \mu and \sigma which in standard form gives respectively 0 and 1. Any normal distribution is a special case of the standard normal distribution N(0,1)

I don't see why using matrix math shouldn't be viewed as a basic primitive. Implementing a matrix library would obscure the point that is being made.

>h = hidden, o = output, y_pred = predicted value of y, etc. are quite clear

  ∀ abbr + to the cogntv load prsn wht mntn in thr hd.

  ∀ -> every
  + -> adds
  wht ->  who has to
  h -> head

Was that easier to read than the following?

  Every abbreviation adds to the cognitive load the person has to maintain in their head.

This isn't code, but the principles of making code (un)readable are exactly the same.

If you see that full-word variable names create too much clutter, it means your code structure is wrong. So you fix it.

You go from this:

    self.w1 = np.random.normal()
    self.w2 = np.random.normal()
    self.w3 = np.random.normal()
    self.w4 = np.random.normal()
    self.w5 = np.random.normal()
    self.w6 = np.random.normal()

To this:

    set_to_random_numbers(self.weights)

After 9 years of maintaining several dozen legacy codebases (which sometimes involved financial math) I am fully convinced that the only people who like terse code are people who write it and never read it after.

That is a good question - I will attempt to answer it for you (probably poorly):

First off, for most problems you don't need a neural network - many, many problems can be solved using simple statistical regression techniques - what could be called "classical machine learning".

It's when your data starts to involve more than a few variables (usually more than 2 or 3) that you need an answer, and/or when you don't know exactly what the best "classical" machine learning technique to use, is when you may find a neural network to be a potential solution.

Ultimately, what a regular neural network does (I'm not going to get into anything like GANs or CNNs or anything special like that - though arguably they all work in a similar manner) is come up with a solution that encapsulates and encodes, through "learning", the proper algorithm(s) needed to provide a solution for the problem - whether that is classification or something else.

It's basically the classic argument that all a neural network is, is a complicated form of linear regression. And to an extent, that is true - but it's done in a "black box" manner, which may or may not be important to you for your problem (the issue becomes "how does it do what it does?" - and that's where things can become tricky to answer).

Neural networks basically take information - various attributes of a problem set - and then, once trained, can output an answer for an example (that it hasn't seen before!) that fits that problem set criteria. It does this by having been "trained" by seeing a lot of different examples, with each example matched up to a "labeled" output. Depending on whether it is right or wrong for the labeled output, it will then propagate the error backwards through the network to correct things by a small amount, and try again (note: your training data needs to be extremely varied, and should include both positive examples and negative examples, and shouldn't be biased toward any particular set of examples).

Anyhow - your data must ultimately be represented by numbers in some manner, usually continuous, and can be anything from a single input to multiple inputs. For instance, you might have the single input of "temperature" to control the output of "turn on/off the furnace" - ie, modeling a thermostat (note that you would never do this actually, outside of learning how a simple perceptron works I guess).

Usually you don't use a single continuous input - you would use multiple inputs - maybe a date, coupled with gps coordinates (like say for housing prices), with an output of "price". Or maybe 10,000 grayscale values (0.00-1.00), representing a 100x100 pixel image, with a classification output of the numbers 0-9 and letters A-Z (36 symbols).

In other words, you are trying to figure out the answer to a problem (is this a picture of a letter or number?) that would be difficult or impossible to code a set of rules by hand to answer with any statistical certainty.

Note that last part: Neural networks do not give you an absolute result; they output a continuous value or a set of values that represent the most probable likelihood of a correct answer based upon what the model has learned in the past. In the case of a single output, that could represent a "yes/no" or "true/false" answer (a value between 1 and 0 respectively); it could also be "left/right", "up/down" or something similar if controlling direction/pressure/flow rate, etc based on inputs.

For classification, it will be a set of continuous values, indicating for each "class" which is the higher probability (0-1) of being correct. For instance, if you had three categories like:

  Monkey: 0.1
  Banana: 0.8
  Coconut: 0.2

Then it is likely that the picture was of a banana. But if you saw this:

  Monkey: 0.7
  Banana: 0.9
  Coconut: 0.1

Maybe the picture contained a monkey holding a banana? It might be possible - even if the network was never trained on that particular imagery!

Note that it is also possible to fool such a network, as numerous studies have shown.

It's also possible to have the network output an image (ie - by having it output a very large array of node values that represents the pixel values of the image); this is how neural networks generate or alter images. Sound data can also be done in a similar manner.

Alright - I think I am rambling a bit, so I'm going to leave it here; I hope this answers at least something of your questions (and I hope it didn't confuse you - if so, I apologize and that wasn't my intent).