These networks ought to be robust to minor changes in W. It's the topology that maters and frankly most of the W_ij != 0 are spurious connections -- meaning perturbation analysis will show that they play no causal role in the computation. I wrote a paper on this which has >100 citation (Survival of The Sparsest: Robust Gene Networks are Parsimonious). I used gene networks, but this is just a special case of neural networks. In fact there been a bunch of papers published on gene regulatory networks that show that topology is the main driver of function -- not surprising, if you show the circuit diagram of an 8-bit adder to an EE, they'll know exactly the function. Logically it has to be so. In fact you can model the gene network of the drosophila segmentation pattern with Boolean (1-bit) networks. The problem with ANN research is that no few take the time to understand why things function as they do. We should be reverse engineering these from biology. Every time a major advancement is made in ANNs neurobiologist say "yes, we could have told you that ten years ago" deep learning is just the latest example. It will hit its asymptote soon, then people will say that AI failed to live up to its expectation, then someone will make a new discovery. It's very frustrating to sit on the sidelines and watch this happen again and again.
Why Are Eight Bits Enough for Deep Neural Networks?
21–30 of 47 posts
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#22I guess this is why rowing crews that regularly practice on choppy water end up doing better in an average competition (no citation, just something my coach once told me). Training in adverse conditions results in better built in corrections.
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#23Earlier quoted context omitted.
Wouldn't noise be a huge issue?
Deep nets love noise, because with so many parameters they are very vulnerable to overfitting. For example, dropout, which is almost ubiquitous for deep learning, basically makes activations 'wrong' 50% of the time during training.
That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect.
Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing.
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#24I had wondered this myself -- it seems reasonable to see limiting the precision of activation as a form of regularization, as the author alludes to. For me, the place we'll eventually end up is obviously custom deep learning / evaluation chips that perform analogue operations using transistors in their linear regime (like how op-amps work). These chips would be programmed merely to express the tensor operation graph,…
I agree that this kind of naive analog computing sounds very attractive with those simple linear operations (linear networks have been exhaustively studied, as you noted you essentially need only resistors and amplifiers). But it's not entirely obvious to me they ought to be better than digital electronics for comparable precision (considering their noise) and power consumption. I think you may get into trouble in th…
you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential
This does not make sense to me. Can you explain?
I think there might be misunderstanding of how analog computing is used to build a neural network. First, a weight is stored as some analog physical property, typically as charge on a floating gate, or on a capacitor in a DRAM type cell. Second, the multiplication operation is performed by modulating the analog input signal going through the floating gate transistor by the charge on the floating gate (weight). Third, the summation is done via simple summation of the currents. Finally, activation function is performed by an opamp.
Regarding power consumption: 1. A digital computer needs a thousand of transistors to perform multiplication, analog circuit can do it with a single one. 2. Analog NN stores parameters (weights) locally, right where they are needed to perform computation. Digital NN will need lots of memory transfers to bring weights from RAM to ALU, and to store intermediate results.
That's why a properly implemented analog NN will always consume much less power.
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#25Earlier quoted context omitted.
I agree that this kind of naive analog computing sounds very attractive with those simple linear operations (linear networks have been exhaustively studied, as you noted you essentially need only resistors and amplifiers). But it's not entirely obvious to me they ought to be better than digital electronics for comparable precision (considering their noise) and power consumption. I think you may get into trouble in th…
> n then there's the fact you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential, so ~n^k vs ~exp(2n) power consumption doesn't look good from this pov. That's true, I feel stupid for not having thought of that! I'm not an electrical engineer, but with the FETs that modern Intel chips…
"parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work!
Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits.
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#26Earlier quoted context omitted.
Deep nets love noise, because with so many parameters they are very vulnerable to overfitting. For example, dropout, which is almost ubiquitous for deep learning, basically makes activations 'wrong' 50% of the time during training.
That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect. Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, ea…
I am pretty sure that parent meant DNNs love to fit noise. So you two are in strong agreement.
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#27These networks ought to be robust to minor changes in W. It's the topology that maters and frankly most of the W_ij != 0 are spurious connections -- meaning perturbation analysis will show that they play no causal role in the computation. I wrote a paper on this which has >100 citation (Survival of The Sparsest: Robust Gene Networks are Parsimonious). I used gene networks, but this is just a special case of neural ne…
Care to do it yourself?
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#28Earlier quoted context omitted.
That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect. Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, ea…
I am pretty sure that parent meant DNNs love to fit noise. So you two are in strong agreement.
No, we are talking about a random electrical circuit noise in the analog NN hardware. Of course, if the noise is known and fixed, the net could learn to compensate (to a certain extent).
The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained.
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#29Eight bits are enough for me … https://www.youtube.com/watch?v=GoGpLl-SUfk
Re: Why Are Eight Bits Enough for Deep Neural Networks?
#30Earlier quoted context omitted.
I agree that this kind of naive analog computing sounds very attractive with those simple linear operations (linear networks have been exhaustively studied, as you noted you essentially need only resistors and amplifiers). But it's not entirely obvious to me they ought to be better than digital electronics for comparable precision (considering their noise) and power consumption. I think you may get into trouble in th…
you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential This does not make sense to me. Can you explain? I think there might be misunderstanding of how analog computing is used to build a neural network. First, a weight is stored as some analog physical property, typically as charge on…
> 1. A digital computer needs a thousand of transistors to perform multiplication, analog circuit can do it with a single one.
That's interesting. What would the circuit be?
> Digital NN will need lots of memory transfers to bring weights from RAM to ALU, and to store intermediate results.
That's not necessarily the case. Cellular neural networks were proposed long ago, for example, and they're digital -- how multiplication happens is independent from the data flow architecture.
> That's why a properly implemented analog NN will always consume much less power.
How do you know that the I^2 cost of operating in the linear regime isn't excessive? I'm totally ignorant on the matter -- I'd love to see a ballpark calculation to understand why it isn't important.