This is really interesting work, but I feel like the way the article summarizes it is getting it wrong. First Single Device To Act Like a Neuron One thing that’s kept engineers from copying the brain’s power efficiency and quirky computational skill is the lack of an electronic device that can, all on its own, act like a neuron. It would take a special kind of device to do that, one whose behavior is more complex tha…
The brain is still way more efficient and parallel then a GPU (e.g. 12W compared to 280W), so being able to duplicate it closer sounds like a pretty compelling advancement.
For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
This runs into the problem that the power efficiency of analogue circuitry is dramatically worse. A FET dissipates the least heat when it's fully on or fully off. Operating in the resistive regime will result in orders of magnitude more dissipation. It's difficult to over-come that, particularly because it's not a comparison between 'analogue v 64-bit float', but 'analogue versus 8-bit int'. (it's tough for scale ana…
That just means an FET isn't the right device for analog circuitry. Maybe this memristor device will be more efficient.
For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
This runs into the problem that the power efficiency of analogue circuitry is dramatically worse. A FET dissipates the least heat when it's fully on or fully off. Operating in the resistive regime will result in orders of magnitude more dissipation. It's difficult to over-come that, particularly because it's not a comparison between 'analogue v 64-bit float', but 'analogue versus 8-bit int'. (it's tough for scale ana…
Ideally each analog device that simulates a neuron is equivalent to tens of thousands of digital transistors that simulate a logical neuron (in space-time for GPUs or CPUs and just space for neuromorphic asics) . This makes the trade-off much more feasible. .
For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
This runs into the problem that the power efficiency of analogue circuitry is dramatically worse. A FET dissipates the least heat when it's fully on or fully off. Operating in the resistive regime will result in orders of magnitude more dissipation. It's difficult to over-come that, particularly because it's not a comparison between 'analogue v 64-bit float', but 'analogue versus 8-bit int'. (it's tough for scale ana…
We don't need to go fully analogue though, going asynchronous might already be a step up. In neural networks some groups do research into so-called spiking neural networks, which you could think of as electronic circuits operating asynchronously on binary signals. They are very energy efficient, but nobody really knows how to use them yet.
For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
> When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number
No, it's O(1). The bit size of how you represent a single datum does not grow with size of your model.
This is really interesting work, but I feel like the way the article summarizes it is getting it wrong. First Single Device To Act Like a Neuron One thing that’s kept engineers from copying the brain’s power efficiency and quirky computational skill is the lack of an electronic device that can, all on its own, act like a neuron. It would take a special kind of device to do that, one whose behavior is more complex tha…
The brain is still way more efficient and parallel then a GPU (e.g. 12W compared to 280W), so being able to duplicate it closer sounds like a pretty compelling advancement.
it also operates at (less than) the speed of sound.
For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
Reminds me of all-optical diffractive deep learning. It's such a cool concept and really made me rethink computing.
For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
> When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number No, it's O(1). The bit size of how you represent a single datum does not grow with size of your model.
That doesn't really contradict what he said. O(n) where n is a positive constant is O(1).
I'm not convinced that spiking behavior is all that's needed to really emulate what's going on in our neurons. Modern hardware designs are not really amenable to replicating the number or structure of synapses that we have, muchless the extremely complicated biochemistry that controls how and when they fire. It's good to see progress being made, but we really need to tone down the hyperbole in the popular scientific press.