Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Are Eight Bits Enough for Deep Neural Networks?

petewarden.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Why Are Eight Bits Enough for Deep Neural Networks?

#31

Post by taliesinb »

p1esk wrote: post #25
taliesinb wrote: post #17

Earlier quoted context omitted.

> n then there's the fact you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential, so ~n^k vs ~exp(2n) power consumption doesn't look good from this pov. That's true, I feel stupid for not having thought of that! I'm not an electrical engineer, but with the FETs that modern Intel chips…

"parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits.

> "parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work!

Of course, but the 'ideal' FET has zero gate capacitance, despite that being the way they work.

> Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits.

Interconnect meaning things like the buses? There's no reason to want a von Neumann architecture for an analog chip. If that leaves leakage, I suppose an analog chip would be the beneficiary of needing a lot fewer transistors per op.

Re: Why Are Eight Bits Enough for Deep Neural Networks?

#32

Post by tachyonbeam »

fgimenez wrote: post #20
LoSboccacc wrote: post #10

did my thesis on this topic (at that time we were searching the lower bound of ALU needed to have them running in zero power devices) it's interesting, NN degrade at about 6bit, and that's mostly because the transfer function become stable and the training gets stuck more often in local minimums. we built a training methodology in two step, first you trained them in 16bit precision, finding the absolute minimum, then…

I had similar empirical results on one of my PhD projects for medical image classification. With small data sets, we got better results on 8-bit data sets compared to 16-bit. We viewed it as a form of regularization that was extremely effective on smaller data sets with a lot of noise (x-rays in this case).

When using 8-bit weights, what kind of mapping do you do? Do you map the 8-bit range into -10 to 10? Do you have more precision near zero or is it a linear mapping?

Re: Why Are Eight Bits Enough for Deep Neural Networks?

#33

Post by taliesinb »

p1esk wrote: post #23
taliesinb wrote: post #18

Earlier quoted context omitted.

Deep nets love noise, because with so many parameters they are very vulnerable to overfitting. For example, dropout, which is almost ubiquitous for deep learning, basically makes activations 'wrong' 50% of the time during training.

That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect. Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, ea…

> Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing.

I was speaking loosely -- dropout is multiplicative Bernoulli noise on the hidden layers.

> That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect.

I'd argue that dropout (and its predecessor in denoising autoencoders) are perfectly valid to see as noise, albeit multiplicative.

Re: Why Are Eight Bits Enough for Deep Neural Networks?

#34

Post by taliesinb »

p1esk wrote: post #28
srean wrote: post #26

Earlier quoted context omitted.

I am pretty sure that parent meant DNNs love to fit noise. So you two are in strong agreement.

No, we are talking about a random electrical circuit noise in the analog NN hardware. Of course, if the noise is known and fixed, the net could learn to compensate (to a certain extent). The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained.

> The noise we are talking about is like when you put your finger on the chip, and raise its temperature by 10 degrees, the whole thing needs to be retrained.

What would change with temperature that would require retraining? Are you saying the output of an op could depend sensitively on temperature, or that higher temperatures would increase things like thermal or shot noise? Why would the latter require retraining?

Re: Why Are Eight Bits Enough for Deep Neural Networks?

#35

Post by taliesinb »

p1esk wrote: post #24
darkmighty wrote: post #6

Earlier quoted context omitted.

I agree that this kind of naive analog computing sounds very attractive with those simple linear operations (linear networks have been exhaustively studied, as you noted you essentially need only resistors and amplifiers). But it's not entirely obvious to me they ought to be better than digital electronics for comparable precision (considering their noise) and power consumption. I think you may get into trouble in th…

you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential This does not make sense to me. Can you explain? I think there might be misunderstanding of how analog computing is used to build a neural network. First, a weight is stored as some analog physical property, typically as charge on…

> This does not make sense to me. Can you explain?

I understood the reasoning to be that to increase the range of accurately representable values in a circuit, you either need to increase the voltage or current used in an analog circuit (to achieve a certain accuracy versus a noise baseline), or devote more bits in a digital circuit. The first gives a linear dependence (or quadratic for I^2 losses) of power on range, the second logarithmic.

Re: Why Are Eight Bits Enough for Deep Neural Networks?

#36

Post by LoSboccacc »

tachyonbeam wrote: post #32
fgimenez wrote: post #20

Earlier quoted context omitted.

I had similar empirical results on one of my PhD projects for medical image classification. With small data sets, we got better results on 8-bit data sets compared to 16-bit. We viewed it as a form of regularization that was extremely effective on smaller data sets with a lot of noise (x-rays in this case).

When using 8-bit weights, what kind of mapping do you do? Do you map the 8-bit range into -10 to 10? Do you have more precision near zero or is it a linear mapping?

Don't know about him but I was working with -8 8 for input and -4 4 for weights, using atan function for transfer maps quite well and there is no need to oversaturate the next layer.

Re: Why Are Eight Bits Enough for Deep Neural Networks?

#37

Post by p1esk »

taliesinb wrote: post #30
p1esk wrote: post #24

Earlier quoted context omitted.

you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential This does not make sense to me. Can you explain? I think there might be misunderstanding of how analog computing is used to build a neural network. First, a weight is stored as some analog physical property, typically as charge on…

> 1. A digital computer needs a thousand of transistors to perform multiplication, analog circuit can do it with a single one. That's interesting. What would the circuit be? > Digital NN will need lots of memory transfers to bring weights from RAM to ALU, and to store intermediate results. That's not necessarily the case. Cellular neural networks were proposed long ago, for example, and they're digital -- how multipl…

As I described above: "the multiplication operation is performed by modulating the analog input signal going through the floating gate transistor by the charge on the floating gate (weight)." The circuit is the single transistor in this case.

Cellular neural networks were proposed long ago, for example, and they're digital

What is so inherently digital about cellular networks? Can you provide a link to an implementation of a cellular net in digital hardware? How the weights are stored? Where the multiplication happens?

Re: Why Are Eight Bits Enough for Deep Neural Networks?

#38

Post by p1esk »

taliesinb wrote: post #35
p1esk wrote: post #24

Earlier quoted context omitted.

you can deal with exponentially larger numbers with roughly linearly (or polynomial) increasing memory, while if you use analog circuits you have to pay a quadratic cost on the exponential This does not make sense to me. Can you explain? I think there might be misunderstanding of how analog computing is used to build a neural network. First, a weight is stored as some analog physical property, typically as charge on…

> This does not make sense to me. Can you explain? I understood the reasoning to be that to increase the range of accurately representable values in a circuit, you either need to increase the voltage or current used in an analog circuit (to achieve a certain accuracy versus a noise baseline), or devote more bits in a digital circuit. The first gives a linear dependence (or quadratic for I^2 losses) of power on range,…

Ah I see. Well, remember, with analog circuits, we are talking about subthreshold currents. This current is orders of magnitude less than the current in a digital circuit (nA vs uA). Correspondingly, the power consumption will be negligible in comparison, even if you expand the current range. And that is only a fraction of the total power consumption. Adding more bits in a digital circuit linearly increases total power, dominated by interconnect capacitance.

Re: Why Are Eight Bits Enough for Deep Neural Networks?

#39

Post by p1esk »

taliesinb wrote: post #31
p1esk wrote: post #25

Earlier quoted context omitted.

"parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits.

> "parasitic gate capacitance" - not sure if you want to call it "parasitic", after all, a gate capacitance is what makes everything work! Of course, but the 'ideal' FET has zero gate capacitance, despite that being the way they work. > Power is mainly lost via leakage (the smaller the transistor, the more it leaks), and via interconnect capacitance, which dominates all other capacitances in modern circuits. Intercon…

'ideal' FET has zero gate capacitance, despite that being the way they work.

I don't understand this statement. What do you mean? A FET is a capacitor (gate to channel). If a gate has no capacitance, you have no transistor.

Interconnect means wire. This has nothing to do with von Neumann architecture. If you have wires in your circuit, then you have wire capacitance. As transistors get smaller, that capacitance starts to dominate internal transistor capacitances.

Re: Why Are Eight Bits Enough for Deep Neural Networks?

#40

Post by p1esk »

taliesinb wrote: post #33
p1esk wrote: post #23

Earlier quoted context omitted.

That's not true. NNs don't like noise, there have been a lot of research done about effect of noise on NNs in the 90s. Random noise over a certain threshold will progressively degrade the performance of NNs, and below the threshold will have no effect. Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, ea…

> Dropout is not the same as random noise. By using dropout you eliminate some neurons from making contribution. As a result, you effectively train many smaller nets, each one adjusting its available weights to perform the same task. During testing, there's no noise - all neurons are back in business and contributing. I was speaking loosely -- dropout is multiplicative Bernoulli noise on the hidden layers. > That's n…

You are missing my point - with dropout, you don't have any noise during the operation of the net. The noise we are talking about (circuit noise) is always present.
Post reply on HN