MNIST Handwritten Digit Classifier – beginner neural network project
1–10 of 31 posts
Re: MNIST Handwritten Digit Classifier – beginner neural network project
#2I thought Neural Networks nowadays use ReLU instead of Sigmoid? Especially in the context of deep learning
Re: MNIST Handwritten Digit Classifier – beginner neural network project
#3Nice. Recreating these methods in simple code for yourself is definitely the way to check you understand it. This demo looks nice, clean, and straightforward. (Although I'd rename or comment variables x and y, or give some sort of guidance on what way around the weight matrices are within the code itself.)
It's also worth checking out existing neural net code-bases to see what tricks they have. The fine details usually aren't in papers, and they're not all in the text-books either.
The first potential problem that jumped out at me in this code was the initialization:
self.weights = [np.array([0])] + [np.random.randn(y, x)
for y, x in zip(sizes[1:], sizes[:-1])]
I'd multiply those initial weights by a small constant divided by the square-root of weights going into the same neuron. For multiple layers you might consider layer-by-layer pre-training. For other architectures, like recurrent nets, definitely find a reference on how to do the initialization.
PS I would definitely add a test routine to check that the gradients from back-propagation agree with a finite difference approximation. It's so easy to get gradient code wrong, and it's so easy to test.
Re: MNIST Handwritten Digit Classifier – beginner neural network project
#4Can someone explain how this repo is so popular/ why it's so popular here? This is a basic implementation of a relatively simple algorithm that you learn when initially starting with ML/DL.
I do not want in any way to sound critical and am genuinely curious about the dynamics of why people would find this interesting given it's reduced complexity.
Re: MNIST Handwritten Digit Classifier – beginner neural network project
#5I thought Neural Networks nowadays use ReLU instead of Sigmoid? Especially in the context of deep learning
Looks like this implementation is based (in part) on the Stanford ML course, which teaches nnets using sigmoid activation.
Given that it's intended to introduce to beginners how nnets work, the choice of activation is an aside anyway - the real meat is back/forwardprop.
Re: MNIST Handwritten Digit Classifier – beginner neural network project
#6Can someone explain how this repo is so popular/ why it's so popular here? This is a basic implementation of a relatively simple algorithm that you learn when initially starting with ML/DL. I do not want in any way to sound critical and am genuinely curious about the dynamics of why people would find this interesting given it's reduced complexity.
You said it yourself: when you want to learn something complex (like ML), it's usually easy to start with something with reduced complexity.
At least, that's why i clicked the link.
Re: MNIST Handwritten Digit Classifier – beginner neural network project
#7Can someone explain how this repo is so popular/ why it's so popular here? This is a basic implementation of a relatively simple algorithm that you learn when initially starting with ML/DL. I do not want in any way to sound critical and am genuinely curious about the dynamics of why people would find this interesting given it's reduced complexity.
One hypothesis is that there is a trend of non ML-familiar developers currently working on getting more grasp on ML. Such repositories provide something that e.g. web developers can take a look at, with reduced friction.
Disclaimer: I'm such a developer! (currently going through the last bits of https://www.coursera.org/learn/machine-learning) - but I've noticed other around me recently.
Re: MNIST Handwritten Digit Classifier – beginner neural network project
#8Nice. Recreating these methods in simple code for yourself is definitely the way to check you understand it. This demo looks nice, clean, and straightforward. (Although I'd rename or comment variables x and y, or give some sort of guidance on what way around the weight matrices are within the code itself.) It's also worth checking out existing neural net code-bases to see what tricks they have. The fine details usual…
'It's also worth checking out existing neural net code-bases to see what tricks they have. The fine details usually aren't in papers, and they're not all in the text-books either.'
Given that you are a person who is highly-qualified to answer, I am genuinely curious why do you think that is? Reimplementing algorithms from scratch is an efficient way to learn, understand the underlying concepts and attempt improvements in a research context.
Re: MNIST Handwritten Digit Classifier – beginner neural network project
#9Nice. Recreating these methods in simple code for yourself is definitely the way to check you understand it. This demo looks nice, clean, and straightforward. (Although I'd rename or comment variables x and y, or give some sort of guidance on what way around the weight matrices are within the code itself.) It's also worth checking out existing neural net code-bases to see what tricks they have. The fine details usual…
[deleted]
Re: MNIST Handwritten Digit Classifier – beginner neural network project
#10I thought Neural Networks nowadays use ReLU instead of Sigmoid? Especially in the context of deep learning
Looks like this implementation is based (in part) on the Stanford ML course, which teaches nnets using sigmoid activation. Given that it's intended to introduce to beginners how nnets work, the choice of activation is an aside anyway - the real meat is back/forwardprop.
Absolutely, though I'll put in softmax, tanh and ReLU activation functions in near time. It isn't that difficult. Putting in more docstrings is also one of my todo.