That Appendix! Next time someone claims people don't have a theoretical understanding of how NNs work point them at that.
And tell them to explain it the rest of us too, since they're so cool and mathy.
Self-Normalizing Neural Networks
11–13 of 13 posts
Re: Self-Normalizing Neural Networks
#12That Appendix! Next time someone claims people don't have a theoretical understanding of how NNs work point them at that.
As I understand it, the theory is not balf of the issue, but the ammount of data processed and generated in training is too much to verify to comprehend manually. So research is currently adding printf debugging, to let the NN explain what it sees, instead of blindly trusting the results.
Re: Self-Normalizing Neural Networks
#13I'm not sure I see why tanh couldn't be used to the same effect. If you use 1.6*tanh(x) as your activation function, it pushes small variances higher and high variances lower and gets you to a variance of ~1 after many layers. Obviously not as rigorous, just an observation.