"We’ll use an optimization algorithm called stochastic gradient descent (SGD) that tells us how to change our weights and biases to minimize loss. It’s basically just this update equation: w1 ← w1 − η ∂w|∂L η is a constant called the learning rate that controls how fast we train. All we’re doing is subtracting η ∂w|∂L from w1: - If ∂L | ∂w1 is positive, w1 will decrease, which makes L decrease - If ∂L | ∂w1 is positi…
I think this comment is facetious (apologies if I’m reading it incorrectly), so I want to offer that this is “simple”, but opaque to someone who is unfamiliar with the syntax or background context. Andrew Ng’s course on machine learning from Stanford (on Coursera, which you should be able to audit for free) gives great intuitive explanations of these topics, if you’re curious enough to invest a couple solid weekends…
I took it in 2011 (ML Class) before Coursera existed, and it finally opened my eyes on not only what and how backprop worked, but also how everything in a NN could be represented and calculated using vectors and matrices (linear algebra), and how that process was "parallelizable".
The course uses Octave as it's programming environment, which is essentially an open-source and (mostly) compatible implementation of Matlab.
My first thought was "Finally! A use case for a home Beowulf cluster that is somewhat practical!"
It really opened my eyes and mind to a number concepts that I had looked into before, but couldn't quite wrap my brain around completely.