Funny that I just finished initial implementation of the code that uses the techniques from the paper (Vanhoucke et al.) mentioned in the post.
Why Are Eight Bits Enough for Deep Neural Networks?
11–20 of 47 posts
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#12Point of interest, if you do the fundamental physics on neuronal membranes, the number of levels that are actually distinguishable give the noise in the system is only about 1000. So even in a biological system there are only 4x the the number of discrete levels. I realize this isn't a good match to what is mentioned in the article but it does put some constraints on the maximum dynamic range that biological sensors…
1024 levels = 10 bits. The article mentions 8 bits, which is 256 levels. Now I get what you mean with 4x.
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#13Agreed. I'm going working on an 8bit floating point that is optimized for learning algos, and optimized to be easy to soft emulate and also very efficient in hardware. One of the cool things about this float is that transfer functions (like the logistic) basically becomes a lookup table for really good performance.
Also, there is no strong need for "zero".
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#14I had wondered this myself -- it seems reasonable to see limiting the precision of activation as a form of regularization, as the author alludes to. For me, the place we'll eventually end up is obviously custom deep learning / evaluation chips that perform analogue operations using transistors in their linear regime (like how op-amps work). These chips would be programmed merely to express the tensor operation graph,…
What are your thoughts on using memristors for neural networks? They appear to have pretty good properties for that.
"Training and Operation of an Integrated Neuromorphic Network Based on Metal-Oxide Memristors", which appeared in Nature recently:
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#15>On the general CPU side, modern SIMD instruction sets are often geared towards float, and so eight bit calculations don’t offer a massive computational advantage on recent x86 or ARM chips.
This isn't true, modern SIMD instruction sets have tons of operations for smaller fixed point numbers, as used heavily in video codecs. Unless the author meant some sort of weird 8 bit float?
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#16I had wondered this myself -- it seems reasonable to see limiting the precision of activation as a form of regularization, as the author alludes to. For me, the place we'll eventually end up is obviously custom deep learning / evaluation chips that perform analogue operations using transistors in their linear regime (like how op-amps work). These chips would be programmed merely to express the tensor operation graph,…
Wouldn't noise be a huge issue?
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#17I had wondered this myself -- it seems reasonable to see limiting the precision of activation as a form of regularization, as the author alludes to. For me, the place we'll eventually end up is obviously custom deep learning / evaluation chips that perform analogue operations using transistors in their linear regime (like how op-amps work). These chips would be programmed merely to express the tensor operation graph,…
I agree that this kind of naive analog computing sounds very attractive with those simple linear operations (linear networks have been exhaustively studied, as you noted you essentially need only resistors and amplifiers). But it's not entirely obvious to me they ought to be better than digital electronics for comparable precision (considering their noise) and power consumption. I think you may get into trouble in th…
> n then there's the fact you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential, so ~n^k vs ~exp(2n) power consumption doesn't look good from this pov.
That's true, I feel stupid for not having thought of that!
I'm not an electrical engineer, but with the FETs that modern Intel chips are using, what fraction of their power consumption comes from parasitic gate capacitance, versus other losses?
And if you operated in the linear region, what's the ballpark steady-state I_SD current you'd need on one FET to drive the gate of the next FET?
I think that's what this comes down to: if gate capacitance dominates other losses even in the linear regime, you still win by not having a clock and lots of digital transitions.
You could even imagine exploiting that: apply 'slow' augmentations of the input data that get you the equivalent of a bunch of iterations on a single example batch, while incurring a much smaller fraction of that initial cost because activations aren't going to change nearly as much as switching to a whole new example batch.
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#18I had wondered this myself -- it seems reasonable to see limiting the precision of activation as a form of regularization, as the author alludes to. For me, the place we'll eventually end up is obviously custom deep learning / evaluation chips that perform analogue operations using transistors in their linear regime (like how op-amps work). These chips would be programmed merely to express the tensor operation graph,…
Wouldn't noise be a huge issue?
Deep nets love noise, because with so many parameters they are very vulnerable to overfitting.
For example, dropout, which is almost ubiquitous for deep learning, basically makes activations 'wrong' 50% of the time during training.
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#19The problem with using digital calculations is that they are deterministic. If a result is really small, it is just rounded down to zero. So if you add a bunch of small numbers, you get zero. Even if the result should be large.
Stochastic rounding can fix this. You round each step with the probability so it's expected value is the same. Usually it will round down to 0, but sometimes it will round up to 1.
Relevant paper, using stochastic rounding. Without it the results get worse and worse before you even get to 8 bits. With stochastic rounding, there is no performance degradation. You could probably even reduce the bits even further. I think it may even be possible to get it down to 1 or 2 bits: http://arxiv.org/abs/1502.02551
The relevant graph: https://i.imgur.com/cOZ4fn3.jpg
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#20did my thesis on this topic (at that time we were searching the lower bound of ALU needed to have them running in zero power devices) it's interesting, NN degrade at about 6bit, and that's mostly because the transfer function become stable and the training gets stuck more often in local minimums. we built a training methodology in two step, first you trained them in 16bit precision, finding the absolute minimum, then…
I had similar empirical results on one of my PhD projects for medical image classification. With small data sets, we got better results on 8-bit data sets compared to 16-bit. We viewed it as a form of regularization that was extremely effective on smaller data sets with a lot of noise (x-rays in this case).