21–30 of 34 posts
This is one cool hack. You can't construct deep neural network from only linear parts because consecutive layers can be always combined into single transformation matrix. That's why you need alternating linear and nonlinear operations. I wonder if it's possible to design special purpose low resolution floating point circuit that maximizes this effect while preserving enough linearity. Then you have fast DNN network p…
? It sounds like the author is ignoring denormals? -edit- Yes, the author is ignoring gradual underflow and the resulting denormal numbers. So as you move from one binate to the next, the spacing between floating point numbers doubles or halves depending on whether you are increasing or decreasing the exponent. When you reach the binate with the most negative possible exponent, you have two choices: a) round toward z…
They added a note at the end clarifying that they have flush to zero mode enabled. quote from the article: "EDIT: This blogpost assumes that we enable flush to zero (FTZ) which treats denormal numbers as zeros. It’d be interesting to see reseachers try without FTZ!"
Evolutionary methods to trap nonlinearities is already hard, I imagine it would be even harder to find functions which exploit even more subtle nonlinearities.
So this exploits the fact that floating point numbers have finite precision (and perhaps uneven spacing) to generate non-linear operations? That's actually a really cool usage of the specification!
Well, except that the author misunderstands the specification and how it is typically implemented on modern computers.
TensorFlow by default is built with denormals off (ftz=true), so denormals aren't relevant for the applications we're interested in. We have updated the post to indicate this — thanks for the feedback!
Earlier quoted context omitted.
In the context of graphics processing that trade-off totally makes sense. Thanks for doing the homework that I was too lazy to do :) It seems to me that in the context of NN computations, using the lack of gradual underflow as a non-linear element is going to severely limit the dynamic range of the neurons. On the plus side, the non-linear element is a computational freebie. But in addition to limited dynamic range,…
Actually if you read section 4.6 of that paper you'll see that denormals are the default on sm_20 and above. But you can see in that same section this this can easily be disabled with the ftz flag. I had to give Jakob custom gemm kernels to do this research. Not sure why the denormal point was left out of this blog as it's pretty critical to the whole experiment.
? It sounds like the author is ignoring denormals? -edit- Yes, the author is ignoring gradual underflow and the resulting denormal numbers. So as you move from one binate to the next, the spacing between floating point numbers doubles or halves depending on whether you are increasing or decreasing the exponent. When you reach the binate with the most negative possible exponent, you have two choices: a) round toward z…
The main problem here is that you're depending on implementation-specific behaviour. If you train on a device, you have to run on a device with exactly the same behaviour. On top of that, some FPUs have very slow (trapping) denormal handling. I'm also unsure how accurate the gradient computation can be when the signal itself has numerical issues. I don't deny it's a cool hack, but beyond that I don't think I see the…
I didn't understand how the gradients are produced to honor this underflow behavior. Is that the reason why they use "ES" instead of symbolic (or probably they meant automatic) differentiation?
It's super interesting to think that any non-linearity at all can make it work. This particular non-linearity is surprising since it's clamping to zero at the center of the response curve. I'd have thought that's right where you want the linear response, and that clamping in the middle would cause bad things to happen. Sigmoid and RelU (and others) clamp at the foot/shoulder. Perhaps this network just learns negative…
This is one cool hack. You can't construct deep neural network from only linear parts because consecutive layers can be always combined into single transformation matrix. That's why you need alternating linear and nonlinear operations. I wonder if it's possible to design special purpose low resolution floating point circuit that maximizes this effect while preserving enough linearity. Then you have fast DNN network p…
At that point, you are probably better off just building circuits with "power this wire to ReLU at the end", which is not very many extra transistors.