What is clear to me is that our current nand based hardware is not ideal to model artificial neurones.
But from my point of view, the difficult part is the many-to-many connectivity.
Memristor Breakthrough: First Single Device to Act Like a Neuron
41–50 of 112 posts
Re: Memristor Breakthrough: First Single Device to Act Like a Neuron
#42Earlier quoted context omitted.
This has nothing to do with "modern neural networks". When they say "neuron" they mean an actual biological neuron. The main application for this device type is brain simulations.
Hasn't this been debunked: https://www.quantamagazine.org/neural-dendrites-reveal-their...
?
Re: Memristor Breakthrough: First Single Device to Act Like a Neuron
#4312 years ago, on HN: "The memristor—the functional equivalent of a synapse—could revolutionize circuit design" https://news.ycombinator.com/item?id=397335
Didn't HP want to build a new type of computer on it?
It was called The Machine: computer with unified memory. They released it without memristors in the end: HPE unveils The Machine, a single-memory computer with 160 terabytes of memory: https://news.ycombinator.com/item?id=14350396
Re: Memristor Breakthrough: First Single Device to Act Like a Neuron
#44Unfortunately, we do not know how neurons learn, compute and represent information in biological neural networks. Though I feel that there would be great potential in figuring this out - the energy efficiency and capabilities of the brain are truly amazing.
Re: Memristor Breakthrough: First Single Device to Act Like a Neuron
#45For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
I totally agree with this. I've long dreamed of something like a FPGA for NN's where the gradient is built into the hardware configuration.
Someday I'll sit down and actually try to design a prototype for such a thing.
Re: Memristor Breakthrough: First Single Device to Act Like a Neuron
#46Earlier quoted context omitted.
This runs into the problem that the power efficiency of analogue circuitry is dramatically worse. A FET dissipates the least heat when it's fully on or fully off. Operating in the resistive regime will result in orders of magnitude more dissipation. It's difficult to over-come that, particularly because it's not a comparison between 'analogue v 64-bit float', but 'analogue versus 8-bit int'. (it's tough for scale ana…
We don't need to go fully analogue though, going asynchronous might already be a step up. In neural networks some groups do research into so-called spiking neural networks, which you could think of as electronic circuits operating asynchronously on binary signals. They are very energy efficient, but nobody really knows how to use them yet.
Clockless CPUs have been a theoretical idea for over a decade before the "neutral network" became a common phrase. Felt like a decade ago it was still a pipe dream. Resolving the asynchronous signals becomes a chore. The world feels stateful and ordered to humans. In the same way functional programming can be difficult, getting this right is very hard.
Re: Memristor Breakthrough: First Single Device to Act Like a Neuron
#47If you find water analogies helpful for circuit elements, here is one for the memristor: "The closest analogy I can think of is a sand filter, an item of apparatus used in water-purification plants. As contaminated water flows through a bed of sand and gravel, sediment gradually clogs the pores of the filter and thereby increases resistance. Reversing the flow flushes out the sediment and reduces resistance." -- http…
Isn’t that like a capacitor? As it charges the voltage increases and the current drops until it reaches zero. Then if you discharge it you can start over again.
Re: Memristor Breakthrough: First Single Device to Act Like a Neuron
#48For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
This runs into the problem that the power efficiency of analogue circuitry is dramatically worse. A FET dissipates the least heat when it's fully on or fully off. Operating in the resistive regime will result in orders of magnitude more dissipation. It's difficult to over-come that, particularly because it's not a comparison between 'analogue v 64-bit float', but 'analogue versus 8-bit int'. (it's tough for scale ana…
> the power efficiency of analogue circuitry is dramatically worse
The human brain would like to have a word with you.
Re: Memristor Breakthrough: First Single Device to Act Like a Neuron
#49If you find water analogies helpful for circuit elements, here is one for the memristor: "The closest analogy I can think of is a sand filter, an item of apparatus used in water-purification plants. As contaminated water flows through a bed of sand and gravel, sediment gradually clogs the pores of the filter and thereby increases resistance. Reversing the flow flushes out the sediment and reduces resistance." -- http…
Isn’t that like a capacitor? As it charges the voltage increases and the current drops until it reaches zero. Then if you discharge it you can start over again.
A capacitor is like a rubber membrane that has elasticity. When the capacitor/rubber membrane is stretched to its limit, it either breaks (dielectric breakdown) or it releases that energy causing a flow in the opposite direction.
A sand filter has no limit (as it gets more clogged, it starts resembling a closed valve) and it can't release built up energy (flowing in the opposite direction just "opens the valve")
Re: Memristor Breakthrough: First Single Device to Act Like a Neuron
#50For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
I totally agree with this. I've long dreamed of something like a FPGA for NN's where the gradient is built into the hardware configuration. Someday I'll sit down and actually try to design a prototype for such a thing.
It's also worth mentioning that the hardware is not the only thing that should change. Cognition seems hard to automate just using instructional directives for producing the states of such a machine. Encoding the gradient right into the hardware would be cool, but the optimization procedure should also move to being something intrinsic to the machine.