If memristors act like neurons, put them in neural networks
spectrum.ieee.org
If memristors act like neurons, put them in neural networks
1–10 of 40 posts
Re: If memristors act like neurons, put them in neural networks
#2Memristors are the technology that seemed poised to usher in a new era of computing. The promise of being able to redraw the current computer architectural hierarchies is tantalizing.
If machine learning applications are what finally get memristors out into the world, I wish them godspeed.
Re: If memristors act like neurons, put them in neural networks
#3I am surprised people have high expectations from memristors. They are just another way to build an analog computer -- better for machine learning, worse for classical ODEs.
But we have not used analog computers for 50 years, and for a good reason -- they are not reproducible, their accuracy is very process dependent and has a hard upper limit, and they are often tuned for a single function.
Would people want a chip which is basically unpredictable -- the performance can vary up by tens of %, they have to be re-trained periodically to prevent data loss, and there is no way to load pre-trained network? I doubt it. Maybe there is an extremely narrow use case, but I do not see it in the mainstream devices.
Re: If memristors act like neurons, put them in neural networks
#4Memristors are the technology that seemed poised to usher in a new era of computing. The promise of being able to redraw the current computer architectural hierarchies is tantalizing. If machine learning applications are what finally get memristors out into the world, I wish them godspeed.
What do you think of memristors vs FPGAs?
They have many similarities -- they both redrew the current computer architectures, they integrate memory and computing, they can have randomness built-in, and they both take less power than mainstream GPUs.
What does memristor provide that specially designed "neural FPGA" can not?
Re: If memristors act like neurons, put them in neural networks
#5I am surprised people have high expectations from memristors. They are just another way to build an analog computer -- better for machine learning, worse for classical ODEs. But we have not used analog computers for 50 years, and for a good reason -- they are not reproducible, their accuracy is very process dependent and has a hard upper limit, and they are often tuned for a single function. Would people want a chip…
>Would people want a chip which is basically unpredictable -- the performance can vary up by tens of %, they have to be re-trained periodically to prevent data loss, and there is no way to load pre-trained network? I doubt it. Maybe there is an extremely narrow use case, but I do not see it in the mainstream devices.
Human brains have the same problems and seem to be fairly popular.
Re: If memristors act like neurons, put them in neural networks
#6Isn't the coherer https://en.wikipedia.org/wiki/Coherer the first memristor? If so it was invented in 1890.
Re: If memristors act like neurons, put them in neural networks
#7I am surprised people have high expectations from memristors. They are just another way to build an analog computer -- better for machine learning, worse for classical ODEs. But we have not used analog computers for 50 years, and for a good reason -- they are not reproducible, their accuracy is very process dependent and has a hard upper limit, and they are often tuned for a single function. Would people want a chip…
You might be interested in 'Robust First Computing' - https://www.youtube.com/watch?v=6CNg1Q3RNWI
edit: here is a far better explanation by the terms creator:https://www.youtube.com/watch?v=7hwO8Q_TyCA
Re: If memristors act like neurons, put them in neural networks
#8Memristors are the technology that seemed poised to usher in a new era of computing. The promise of being able to redraw the current computer architectural hierarchies is tantalizing. If machine learning applications are what finally get memristors out into the world, I wish them godspeed.
What do you think of memristors vs FPGAs? They have many similarities -- they both redrew the current computer architectures, they integrate memory and computing, they can have randomness built-in, and they both take less power than mainstream GPUs. What does memristor provide that specially designed "neural FPGA" can not?
They aren't really directly comparable like you imply. A memristor is basically a resistor that changes resistance when you run current through it. An FPGA is a clever layout of many thousands of logic gates that can be programmed to form arbitrarily complex digital logic circuits. It's like comparing apples to skyscrapers because they're both associated with New York.
Presumably, a memristor based neural network would have the advantage over an FPGA of requiring significantly less silicon area to achieve the same function. I imagine an FPGA based neural network would approximate analog signals digitally, perhaps using floating point "half's" or something. Memristors would directly operate on analog signals, encoding information as amplitudes or pulses of currents and voltages.
Notably, FPGAs and GPUs can't really be directly compared to each other in terms of power consumption unless you specify specific use cases. You can't build the equivalent of a mainstream GPU out of FPGAs without severely limiting the clock speed (because of how physically large it would be), and if you did anyway, it would use many orders of magnitude more power to function. So, a GPU is way more power efficient than an FPGA for rendering graphics. There are certainly problems that a GPU isn't good at solving, and so there's a good chance that an FPGA solution would be more power efficient.
Re: If memristors act like neurons, put them in neural networks
#9I am surprised people have high expectations from memristors. They are just another way to build an analog computer -- better for machine learning, worse for classical ODEs. But we have not used analog computers for 50 years, and for a good reason -- they are not reproducible, their accuracy is very process dependent and has a hard upper limit, and they are often tuned for a single function. Would people want a chip…
I agree with all your points. Perhaps the opportunity is if your neural networks are tolerant to that kind of variance there may be applications where you can shrink the hardware footprint (and cost) dramatically while improving latencies (and maybe power usage).
Re: If memristors act like neurons, put them in neural networks
#10Earlier quoted context omitted.
What do you think of memristors vs FPGAs? They have many similarities -- they both redrew the current computer architectures, they integrate memory and computing, they can have randomness built-in, and they both take less power than mainstream GPUs. What does memristor provide that specially designed "neural FPGA" can not?
They aren't really directly comparable like you imply. A memristor is basically a resistor that changes resistance when you run current through it. An FPGA is a clever layout of many thousands of logic gates that can be programmed to form arbitrarily complex digital logic circuits. It's like comparing apples to skyscrapers because they're both associated with New York. Presumably, a memristor based neural network wou…
Correct me if I'm wrong, but isn't it a good metaphor to say that FPGAs are just ASIC emulators? I.e. much less efficient than an equivalent ASIC, but good for prototyping and something to use in production if you can't afford to manufacture your own chip (which is most low-volume use cases). Under this metaphor, memristor-based ASICs will obviously be more efficient than FPGAs emulating them, but that's predicated on there being a high-volume use case justifying the creation of a memristor chip in the first place.