Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nonlinear Computation in Deep Linear Networks

blog.openai.com

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Nonlinear Computation in Deep Linear Networks

#11

Post by Plough_Jogger »

BrianMingus wrote: post #8

These findings seem to be at odds. The former says that deep linear nets are useful, non-linear and trainable with gradient descent. The latter says that the non-linearity only exists due to quirks in floating point and that evolutionary strategies must be use to find extremely small activations that can exploit the non-linearities in floating point. Exact solutions to the nonlinear dynamics of learning in deep linea…

The arxiv paper here is analyzing the nonlinearities in a network's learning dynamics; exploring why training time / error rates are not do not vary linearly throughout the the training process.

They note: "Here we provide an exact analytical theory of learning in deep linear neural networks that quantitatively answers these questions for this restricted setting. Because of its linearity, the input-output map of a deep linear network can always be rewritten as a shallow network."

Re: Nonlinear Computation in Deep Linear Networks

#12

Post by Nokinside »

This is one cool hack.

You can't construct deep neural network from only linear parts because consecutive layers can be always combined into single transformation matrix. That's why you need alternating linear and nonlinear operations.

I wonder if it's possible to design special purpose low resolution floating point circuit that maximizes this effect while preserving enough linearity. Then you have fast DNN network pipeline constructed from just summation and addition.

Re: Nonlinear Computation in Deep Linear Networks

#13

Post by aray »

dbcurtis wrote: post #3

? It sounds like the author is ignoring denormals? -edit- Yes, the author is ignoring gradual underflow and the resulting denormal numbers. So as you move from one binate to the next, the spacing between floating point numbers doubles or halves depending on whether you are increasing or decreasing the exponent. When you reach the binate with the most negative possible exponent, you have two choices: a) round toward z…

It looks like nvidia GPUs treat denormals as zeros for single-precision floating point math: http://developer.download.nvidia.com/assets/cuda/files/NVIDI... (sections 4.1 and 4.2)

Re: Nonlinear Computation in Deep Linear Networks

#14

Post by dbcurtis »

aray wrote: post #13
dbcurtis wrote: post #3

? It sounds like the author is ignoring denormals? -edit- Yes, the author is ignoring gradual underflow and the resulting denormal numbers. So as you move from one binate to the next, the spacing between floating point numbers doubles or halves depending on whether you are increasing or decreasing the exponent. When you reach the binate with the most negative possible exponent, you have two choices: a) round toward z…

It looks like nvidia GPUs treat denormals as zeros for single-precision floating point math: http://developer.download.nvidia.com/assets/cuda/files/NVIDI... (sections 4.1 and 4.2)

In the context of graphics processing that trade-off totally makes sense.

Thanks for doing the homework that I was too lazy to do :)

It seems to me that in the context of NN computations, using the lack of gradual underflow as a non-linear element is going to severely limit the dynamic range of the neurons. On the plus side, the non-linear element is a computational freebie. But in addition to limited dynamic range, it makes the NN ridiculously non-portable across hardware implementations.

Re: Nonlinear Computation in Deep Linear Networks

#16

Post by joe_the_user »

dbcurtis wrote: post #3

? It sounds like the author is ignoring denormals? -edit- Yes, the author is ignoring gradual underflow and the resulting denormal numbers. So as you move from one binate to the next, the spacing between floating point numbers doubles or halves depending on whether you are increasing or decreasing the exponent. When you reach the binate with the most negative possible exponent, you have two choices: a) round toward z…

I'm not sure why you say "ignore".

As I read this, the author claims to have created a naive "linear" network akin regular deep learning networks but without the added (explicitly) non-linearity and shows it's trainable. He acknowledges it has to operate through non-linearity (indeed underflow) and so the mechanisms you mention sound compatible with his findings.

The point I'd see for the article isn't some magic non-linear to linear transformation but that for all we know, incidental underflow effects might operating in regular "non-linear" networks as well.

Re: Nonlinear Computation in Deep Linear Networks

#17

Post by jmvalin »

dbcurtis wrote: post #3

? It sounds like the author is ignoring denormals? -edit- Yes, the author is ignoring gradual underflow and the resulting denormal numbers. So as you move from one binate to the next, the spacing between floating point numbers doubles or halves depending on whether you are increasing or decreasing the exponent. When you reach the binate with the most negative possible exponent, you have two choices: a) round toward z…

The main problem here is that you're depending on implementation-specific behaviour. If you train on a device, you have to run on a device with exactly the same behaviour. On top of that, some FPUs have very slow (trapping) denormal handling. I'm also unsure how accurate the gradient computation can be when the signal itself has numerical issues.

I don't deny it's a cool hack, but beyond that I don't think I see the point or the problem this is trying to solve.

Re: Nonlinear Computation in Deep Linear Networks

#19

Post by scott-gray »

dbcurtis wrote: post #14
aray wrote: post #13

Earlier quoted context omitted.

It looks like nvidia GPUs treat denormals as zeros for single-precision floating point math: http://developer.download.nvidia.com/assets/cuda/files/NVIDI... (sections 4.1 and 4.2)

In the context of graphics processing that trade-off totally makes sense. Thanks for doing the homework that I was too lazy to do :) It seems to me that in the context of NN computations, using the lack of gradual underflow as a non-linear element is going to severely limit the dynamic range of the neurons. On the plus side, the non-linear element is a computational freebie. But in addition to limited dynamic range,…

Actually if you read section 4.6 of that paper you'll see that denormals are the default on sm_20 and above. But you can see in that same section this this can easily be disabled with the ftz flag.

I had to give Jakob custom gemm kernels to do this research. Not sure why the denormal point was left out of this blog as it's pretty critical to the whole experiment.

Re: Nonlinear Computation in Deep Linear Networks

#20

Post by zaphos »

dbcurtis wrote: post #3

? It sounds like the author is ignoring denormals? -edit- Yes, the author is ignoring gradual underflow and the resulting denormal numbers. So as you move from one binate to the next, the spacing between floating point numbers doubles or halves depending on whether you are increasing or decreasing the exponent. When you reach the binate with the most negative possible exponent, you have two choices: a) round toward z…

They added a note at the end clarifying that they have flush to zero mode enabled.

quote from the article: "EDIT: This blogpost assumes that we enable flush to zero (FTZ) which treats denormal numbers as zeros. It’d be interesting to see reseachers try without FTZ!"

Post reply on HN