This runs into the problem that the power efficiency of analogue circuitry is dramatically worse. A FET dissipates the least heat when it's fully on or fully off. Operating in the resistive regime will result in orders of magnitude more dissipation. It's difficult to over-come that, particularly because it's not a comparison between 'analogue v 64-bit float', but 'analogue versus 8-bit int'. (it's tough for scale ana…
> the power efficiency of analogue circuitry is dramatically worse The human brain would like to have a word with you.
At the slow "clock speed" the brain runs at, it produces a lot of heat. A quick search turns up 10-20 watts as a rough estimate. It's still petty competitive given what it's capable of.
For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
If you're building adders with O(n) carry times you're doing it wrong - normal carry lookahead logic is ~O(log N)
12 years ago, on HN: "The memristor—the functional equivalent of a synapse—could revolutionize circuit design" https://news.ycombinator.com/item?id=397335
It's fascinating that none of the comments mention neural network applications.
Seems like these people came a bit too early to the party.
This runs into the problem that the power efficiency of analogue circuitry is dramatically worse. A FET dissipates the least heat when it's fully on or fully off. Operating in the resistive regime will result in orders of magnitude more dissipation. It's difficult to over-come that, particularly because it's not a comparison between 'analogue v 64-bit float', but 'analogue versus 8-bit int'. (it's tough for scale ana…
That just means an FET isn't the right device for analog circuitry. Maybe this memristor device will be more efficient.
It's a bit more fundamental than that. A FET goes between say, 0.01 Ohms and 1,000,000 Ohms for fully on and fully off. The times in between are where it passes the most power through it and that's because of how Ohms law and Power being current times voltage (ignoring differing phase between current and voltage in AC circuits). This all still remains true even for the memristors. You can do analog stuff with higher resistance and lower currents, but then you end up more susceptible to noise and other randomness induced by the environment: thermal noise, electric noise, magnetic noise, and RF noise (technically also electric noise, but more organized), etc. All that being said, for a NN this might actually be somewhat acceptable, but for general analog circuitry you can't get past this problem at all. That's a large reason why most analog circuitry these days does as much as possible in a digital domain before converting back to analog, even simple amplification. A class D amplifier does the whole job by chopping up a large voltage into just on and off and then filters it afterwards. Even though you have power loss in the filtering, the end result is much easier to have stable, and get the desired amplification. You still find class A and B amplifiers where the high frequency chopping noise isn't possible to filter out or would cause other problems, particularly when you look at RF circuitry since you'd end up transmitting that high frequency noise and causing other problems for yourself.
For neural networks, it makes sense to skip pure digital design. When I learned how to design an ALU to say, add, and wait for the propagation of carry bits that’s like O(n) where n is the number of bits in the number, it made me want to just use superposition for addition, which is physical and instantaneous. Of course, that has all sorts of other problems that make it worse (so much worse). Once you learn how to sl…
A bottom-up rewrite of modern computer engineering for performance and capability would probably qualify as one of our largest engineering endeavors in human history. But I would totally be on board for that. There would inevitably be some really interesting discoveries from it, no doubt.
> the power efficiency of analogue circuitry is dramatically worse The human brain would like to have a word with you.
At the slow "clock speed" the brain runs at, it produces a lot of heat. A quick search turns up 10-20 watts as a rough estimate. It's still petty competitive given what it's capable of.
Off topic. When something is on the tip of my tongue, I don't search for it because I have a strange belief that what my mind is trying to do to remember it is fire off neurons associated with the thought or idea, near it, to create the action potential. It is strange that I will eventually remember it within hours or the next day. When the action potential fires the neuron, dendrites are formed between neurons strengthening the memory.
I wonder if there is any truth to that notion? That to solidify a memory, there can't be outside help, that electric chemical signal has to trigger firing the neurons associated with the memory to strengthen their connections.