Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural network chip built using memristors

arstechnica.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Neural network chip built using memristors

#31

Post by p1esk »

modeless wrote: post #30
meric wrote: post #15

A neural network chip semi-conductor startup: http://brainchipinc.com/technology/ They "backdoor listed" on to an Australian mining company, share price went from 1 cent to 27 cents: https://www.google.com/finance?cid=11163357 Valued at $57m.

Man, why are all the silicon people so fixated on spiking nets? Maybe in 20 years once we figure out how the brain works they'll be great but if you built a convnet chip instead then it could be smashing records in real problems like speech recognition, translation, image identification, etc, today . Where are the convnet chip startups?

First of all, what makes you think a hardware convnet will perform better than a software convnet?

Second, you can implement a convnet with a spiking circuit: http://link.springer.com/article/10.1007%2Fs11263-014-0788-3

Re: Neural network chip built using memristors

#32

Post by modeless »

p1esk wrote: post #31
modeless wrote: post #30

Earlier quoted context omitted.

Man, why are all the silicon people so fixated on spiking nets? Maybe in 20 years once we figure out how the brain works they'll be great but if you built a convnet chip instead then it could be smashing records in real problems like speech recognition, translation, image identification, etc, today . Where are the convnet chip startups?

First of all, what makes you think a hardware convnet will perform better than a software convnet? Second, you can implement a convnet with a spiking circuit: http://link.springer.com/article/10.1007%2Fs11263-014-0788-3

That paper does not implement training, only testing. Low-power testing is good to have e.g. for mobile applications, but it doesn't increase the capabilities of our algorithms. What we need to advance the field is faster training of larger nets. That's where the really interesting applications will be discovered.

A convnet-optimized chip could clearly be much faster and more power efficient than a convnet running on a CPU or GPU. The move from CPU to GPU brought a 10x speedup already, but GPUs are hardly ideal for running convnets. For one thing, they have tons of graphics-specific hardware that's useless for convnets and could just be deleted in a convnet chip. For another, GPUs are much more flexible than necessary for convnets. The main operation you need to perform is convolution and you could make fixed-function convolution units that would be much more power and area efficient than generalized GPU shader cores. For yet another thing, there's no reason to believe that 32-bit IEEE 754 floating point is the best power/precision tradeoff for convnets. I'm willing to bet that you could go much lower. You could even experiment with approximate arithmetic; 0.5 ULP precision is probably not necessary.

Re: Neural network chip built using memristors

#33

Post by modeless »

p1esk wrote: post #20
chimtim wrote: post #19

This looks cool but I'm somewhat skeptical. I would be more interested in seeing what problem the system solves better or decently (say even MNIST) rather than how it was built using memristors. There is a lesson from IBM trying to mimic a rat's brain -- that is you try to solve a problem rather than just burn power.

>>>I would be more interested in seeing what problem the system solves better Better than what?

Better than previous state of the art on a standard machine learning dataset. Check some leaderboards here: http://rodrigob.github.io/are_we_there_yet/build/#datasets

Re: Neural network chip built using memristors

#34

Post by e12e »

thangalin wrote: post #27
andyl wrote: post #4

"Even on a 30 nm process, it would be possible to place 25 million cells in a square centimeter, with 10,000 synapses on each cell. And all that would dissipate about a Watt." Wow - seems like a lot. Human brain by comparison (sourced by google): - 12 watts - 100 billion neurons - 1000 trillion connections Computing with memsisters is going to be very interesting.

The human brain has between 100 and 500 trillion synapses and consumes a lowly 12 watts. (In contrast, a 12.6 megawatt supercomputer, in 2013, took 40 minutes to simulate one second of biological brain activity.) The article cites 250 billion synapses per watt. For the same 12 watts as a human brain eats up, a set of memristors could simulate three trillion synapses. A cat, in comparison, has 10 trillion. To get 100…

The thought of a drone with the intelligence of a cat is a scary thought...

The numbers are interesting, though. 400 square centimetres sounds to me to be in the ballpark of a human brain (accounting for several layers).

Re: Neural network chip built using memristors

#36

Post by vegedor »

p1esk wrote: post #20
chimtim wrote: post #19

This looks cool but I'm somewhat skeptical. I would be more interested in seeing what problem the system solves better or decently (say even MNIST) rather than how it was built using memristors. There is a lesson from IBM trying to mimic a rat's brain -- that is you try to solve a problem rather than just burn power.

>>>I would be more interested in seeing what problem the system solves better Better than what?

better than anything else, obviously.

Re: Neural network chip built using memristors

#37

Post by p1esk »

modeless wrote: post #32
p1esk wrote: post #31

Earlier quoted context omitted.

First of all, what makes you think a hardware convnet will perform better than a software convnet? Second, you can implement a convnet with a spiking circuit: http://link.springer.com/article/10.1007%2Fs11263-014-0788-3

That paper does not implement training, only testing. Low-power testing is good to have e.g. for mobile applications, but it doesn't increase the capabilities of our algorithms. What we need to advance the field is faster training of larger nets. That's where the really interesting applications will be discovered. A convnet-optimized chip could clearly be much faster and more power efficient than a convnet running on…

Building a hardware convnet is only beneficial when you figured out the exact parameters of the network. Everything is hardwired. Therefore, it's useless if you want to experiment with lots of different parameters, tricks, or architectures to "advance the field".

Moreover, building such a chip is an expensive and long process, and given how fast GPUs are improving, it's not clear that by the time you build it, it will still be competitive.

Finally, if you want to speed up training, try to figure out a better algorithm. For example, humans can learn from very few training examples. Current neural networks need many thousands, or even millions. There's a potential million-fold speed up in training time right here - and to find it, you need the flexibility of the software.

Re: Neural network chip built using memristors

#38

Post by p1esk »

modeless wrote: post #33
p1esk wrote: post #20

Earlier quoted context omitted.

>>>I would be more interested in seeing what problem the system solves better Better than what?

Better than previous state of the art on a standard machine learning dataset. Check some leaderboards here: http://rodrigob.github.io/are_we_there_yet/build/#datasets

What are you talking about? They build a 12x12 crossbar. The best you can do with it is to implement a single layer perceptron to classify 3x3 pixel patterns. Once they figure out how to scale it up, they will implement a larger network.

Re: Neural network chip built using memristors

#39

Post by JulianMorrison »

themeek wrote: post #11

This is one of the exotic devices in DARPA's UPSIDE competition for exascale computing. This initiative seeks to find non-state (non-transistor) based approaches to computation: exploitation of nanoscale response properties of discrete components to perform some restricted, non-binary, forms of computation. Essentially, exotic ways to abuse silicon lithography to get analog computation. The idea, and this can be seen…

So, Skynet, basically?

Re: Neural network chip built using memristors

#40

Post by neolefty »

thangalin wrote: post #27
andyl wrote: post #4

"Even on a 30 nm process, it would be possible to place 25 million cells in a square centimeter, with 10,000 synapses on each cell. And all that would dissipate about a Watt." Wow - seems like a lot. Human brain by comparison (sourced by google): - 12 watts - 100 billion neurons - 1000 trillion connections Computing with memsisters is going to be very interesting.

The human brain has between 100 and 500 trillion synapses and consumes a lowly 12 watts. (In contrast, a 12.6 megawatt supercomputer, in 2013, took 40 minutes to simulate one second of biological brain activity.) The article cites 250 billion synapses per watt. For the same 12 watts as a human brain eats up, a set of memristors could simulate three trillion synapses. A cat, in comparison, has 10 trillion. To get 100…

Does that take into account computation speed per synapse/memristor?
Post reply on HN