Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hardware Architectures for Deep Neural Networks [pdf]

rle.mit.edu

Post reply on HN
Print view View original

11–19 of 19 posts

Re: Hardware Architectures for Deep Neural Networks [pdf]

#11

Post by skummetmaelk »

neolefty wrote: post #6

I've been wanting to know how electronics' power efficiency compares to biological neurons, and this paper gives a clue. The most efficient hardware it mentions is the GV100 "Tensor Core", at 400GFLOPS/W for FP16. If a typical neuron requires 10^6 ATP per activation [1], and if it takes 30.5 kJ/mol to charge ATP [2], and if a typical neuron has 100 axons, each of which is contributing one FLOP, then I _think_ a human…

A neuron has only 1 axon, it may branch to many different places however.

It's more useful to loop at synaptic operations. 1 FLOP per synaptic activation is a vast underestimate of what happens computationally in the brain. Synaptic activations have many homeostatic side effects.

One can also not discount the cost of the memory needed for the tensor core.

Re: Hardware Architectures for Deep Neural Networks [pdf]

#12

Post by neolefty »

deepnotderp wrote: post #9
neolefty wrote: post #6

I've been wanting to know how electronics' power efficiency compares to biological neurons, and this paper gives a clue. The most efficient hardware it mentions is the GV100 "Tensor Core", at 400GFLOPS/W for FP16. If a typical neuron requires 10^6 ATP per activation [1], and if it takes 30.5 kJ/mol to charge ATP [2], and if a typical neuron has 100 axons, each of which is contributing one FLOP, then I _think_ a human…

Well, keep in mind that you should never really use the advertised number as your comparison point, because that's more the "highest that you'll ever see" rather than the "average reliable performance". Not to mention they assume everything magically is in the caches as far as memory is concerned, which is absolutely untrue.

Yes, there are many confounding factors and sources of uncertainty on both sides -- here are a few I can think of:

* memory isn't accounted for

* FLOP per activation is estimated at 100

* neurons can vary by a factor of ~1000 in how much energy they require to activate, based on their size and speed

* ATP: I'm not really sure what to make of the energy here; I used 30kJ / mol. Wikipedia says "The hydrolysis of ATP into ADP and inorganic phosphate releases 30.5 kJ/mol of enthalpy, with a change in free energy of 3.4 kJ/mol."[1] What's the efficiency of our digestive & recharging systems -- maybe 5% - 30%?

https://en.wikipedia.org/wiki/Adenosine_triphosphate

Re: Hardware Architectures for Deep Neural Networks [pdf]

#13

Post by TeMPOraL »

bga wrote: post #7
neolefty wrote: post #6

I've been wanting to know how electronics' power efficiency compares to biological neurons, and this paper gives a clue. The most efficient hardware it mentions is the GV100 "Tensor Core", at 400GFLOPS/W for FP16. If a typical neuron requires 10^6 ATP per activation [1], and if it takes 30.5 kJ/mol to charge ATP [2], and if a typical neuron has 100 axons, each of which is contributing one FLOP, then I _think_ a human…

Keep in mind the energy costs of the supporting hardware for that brain, though.

Computers have supporting hardware too - powerplants (plus related infrastructure) and cooling systems, at minimum.

Re: Hardware Architectures for Deep Neural Networks [pdf]

#14

Post by 0xbear »

neolefty wrote: post #6

I've been wanting to know how electronics' power efficiency compares to biological neurons, and this paper gives a clue. The most efficient hardware it mentions is the GV100 "Tensor Core", at 400GFLOPS/W for FP16. If a typical neuron requires 10^6 ATP per activation [1], and if it takes 30.5 kJ/mol to charge ATP [2], and if a typical neuron has 100 axons, each of which is contributing one FLOP, then I _think_ a human…

How do you figure an "axon contributes one FLOP" if it's spiking? Seems to me a _spike train_ (that is, a long series of activations) would be equivalent to a floating point operation, not a single spike.

Re: Hardware Architectures for Deep Neural Networks [pdf]

#15

Post by bobsil1 »

neolefty wrote: post #6

I've been wanting to know how electronics' power efficiency compares to biological neurons, and this paper gives a clue. The most efficient hardware it mentions is the GV100 "Tensor Core", at 400GFLOPS/W for FP16. If a typical neuron requires 10^6 ATP per activation [1], and if it takes 30.5 kJ/mol to charge ATP [2], and if a typical neuron has 100 axons, each of which is contributing one FLOP, then I _think_ a human…

Human brain uses 10^5 less power than von Neumann architecture, according to Caltech paper from ‘90s.

Re: Hardware Architectures for Deep Neural Networks [pdf]

#16

Post by mannigfaltig »

TeMPOraL wrote: post #13
bga wrote: post #7

Earlier quoted context omitted.

Keep in mind the energy costs of the supporting hardware for that brain, though.

Computers have supporting hardware too - powerplants (plus related infrastructure) and cooling systems, at minimum.

Humans consume 0.7-1.3 kW per capita. A network of these chips scaled to human brain capacity would consume around 10 kW. It is really not a big difference. Our robot overlords are going to be quite efficient!

Re: Hardware Architectures for Deep Neural Networks [pdf]

#17

Post by metaphor »

bobsil1 wrote: post #15
neolefty wrote: post #6

I've been wanting to know how electronics' power efficiency compares to biological neurons, and this paper gives a clue. The most efficient hardware it mentions is the GV100 "Tensor Core", at 400GFLOPS/W for FP16. If a typical neuron requires 10^6 ATP per activation [1], and if it takes 30.5 kJ/mol to charge ATP [2], and if a typical neuron has 100 axons, each of which is contributing one FLOP, then I _think_ a human…

Human brain uses 10^5 less power than von Neumann architecture, according to Caltech paper from ‘90s.

Do you have a specific reference?

This Mead paper[1] suggests "...a factor of 10 million more efficient than the best digital technology that we can imagine"...which I interpreted as a gross approximation at the chip level assuming 10nm process, not considering any particular architecture, and doesn't include roughly 2-3 orders of magnitude reduced efficient at the system level.

[1] http://authors.library.caltech.edu/53090/1/00058356.pdf

Re: Hardware Architectures for Deep Neural Networks [pdf]

#18

Post by bobsil1 »

metaphor wrote: post #17
bobsil1 wrote: post #15

Earlier quoted context omitted.

Human brain uses 10^5 less power than von Neumann architecture, according to Caltech paper from ‘90s.

Do you have a specific reference? This Mead paper[1] suggests "...a factor of 10 million more efficient than the best digital technology that we can imagine"...which I interpreted as a gross approximation at the chip level assuming 10nm process, not considering any particular architecture, and doesn't include roughly 2-3 orders of magnitude reduced efficient at the system level. [1] http://authors.library.caltech.edu…

That was the paper I was thinking of (thanks), but maybe the figure is from an IBM True North paper. I vaguely recall 10^-5 for brain efficiency, 10^-4 for True North vs. von Neumann.

Re: Hardware Architectures for Deep Neural Networks [pdf]

#19

Post by neolefty »

0xbear wrote: post #14
neolefty wrote: post #6

I've been wanting to know how electronics' power efficiency compares to biological neurons, and this paper gives a clue. The most efficient hardware it mentions is the GV100 "Tensor Core", at 400GFLOPS/W for FP16. If a typical neuron requires 10^6 ATP per activation [1], and if it takes 30.5 kJ/mol to charge ATP [2], and if a typical neuron has 100 axons, each of which is contributing one FLOP, then I _think_ a human…

How do you figure an "axon contributes one FLOP" if it's spiking? Seems to me a _spike train_ (that is, a long series of activations) would be equivalent to a floating point operation, not a single spike.

My estimate is undoubtedly a case of "Doesn't realize the difficulty of the question, therefore naively tries to answer it."

I think you have a much more sophisticated understanding of neurons than I do!

Post reply on HN