Nonlinear Computation in Deep Linear Networks
blog.openai.com
Nonlinear Computation in Deep Linear Networks
1–10 of 34 posts
Re: Nonlinear Computation in Deep Linear Networks
#2Wasn't this already obvious from simple networks which implement e.g. the XOR function?
Re: Nonlinear Computation in Deep Linear Networks
#3? It sounds like the author is ignoring denormals?
-edit-
Yes, the author is ignoring gradual underflow and the resulting denormal numbers.
So as you move from one binate to the next, the spacing between floating point numbers doubles or halves depending on whether you are increasing or decreasing the exponent. When you reach the binate with the most negative possible exponent, you have two choices: a) round toward zero, which leads to a huge non-monotonic jump in the spacing of numbers on the floating point number line. This is a great annoyance to numerical analysts and leads to convergence instabilities. That is why any modern computer used for numerical work incorporates choice b) gradual underflow, which implies that you must allow non-normalized numbers in the two binates (the two being + and - sign bit) of the most negative exponent, which has the effect of creating another pair of binates around zero. This keeps the spacing of numbers on the floating point number line the same in the four binates around zero. Numerical algorithms are then much more stable.
I haven't looked at what GPU's do, I strongly suspect that they round toward zero, because first of all it doesn't matter much to graphics applications, and secondly, the typical method of handling denormals is to take a trap and drop into software emulated floating point because the cost of the additional hardware to handle denormals is very large and the hardware complexity is crazy-making. A GPU isn't going to want to break the pipeline for a denormal.
Re: Nonlinear Computation in Deep Linear Networks
#4Wasn't this already obvious from simple networks which implement e.g. the XOR function?
This is completely different. It seems to be about exploiting floating point underflow to have linear nodes perform nonlinear computation.
Re: Nonlinear Computation in Deep Linear Networks
#5I guess the hope is that a round of computation can be shaved off.
Re: Nonlinear Computation in Deep Linear Networks
#6Wasn't this already obvious from simple networks which implement e.g. the XOR function?
Simple network without non linear function (eg sigmoid) can’t learn XOR
Re: Nonlinear Computation in Deep Linear Networks
#7So this exploits the fact that floating point numbers have finite precision (and perhaps uneven spacing) to generate non-linear operations?
That's actually a really cool usage of the specification!
Re: Nonlinear Computation in Deep Linear Networks
#8These findings seem to be at odds. The former says that deep linear nets are useful, non-linear and trainable with gradient descent. The latter says that the non-linearity only exists due to quirks in floating point and that evolutionary strategies must be use to find extremely small activations that can exploit the non-linearities in floating point.
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
https://arxiv.org/abs/1312.6120
"We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions."
Nonlinear Computation in Deep Linear Networks
https://blog.openai.com/nonlinear-computation-in-linear-netw...
"Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear layer. So it’s a surprise that floating point arithmetic is nonlinear enough to yield trainable deep networks."
Re: Nonlinear Computation in Deep Linear Networks
#9So this exploits the fact that floating point numbers have finite precision (and perhaps uneven spacing) to generate non-linear operations? That's actually a really cool usage of the specification!
Well, except that the author misunderstands the specification and how it is typically implemented on modern computers.
Re: Nonlinear Computation in Deep Linear Networks
#10These findings seem to be at odds. The former says that deep linear nets are useful, non-linear and trainable with gradient descent. The latter says that the non-linearity only exists due to quirks in floating point and that evolutionary strategies must be use to find extremely small activations that can exploit the non-linearities in floating point. Exact solutions to the nonlinear dynamics of learning in deep linea…
[deleted]