Event-based backpropagation for exact gradients in spiking neural networks
1–10 of 38 posts
Re: Event-based backpropagation for exact gradients in spiking neural networks
#2This is about achieving Deep learning on Neuromorphic hardware. Large research teams have been working on it for decades. Billions of dollars/Euros/Pounds must have been poured into it. Still, their devices and algorithms get blown out of the water by an off-the-shelf GPU plus tensorflow, pytorch, what have you.
Hats off for the authors' achievement, this is no small feat and something that has been tried for years. But IMHO it's time that field moved on from running after matrix accelerators and focused on the real advantages of event-based computing: asynchronous, low-latency, event-based signal processing.
Re: Event-based backpropagation for exact gradients in spiking neural networks
#3This is about achieving Deep learning on Neuromorphic hardware. Large research teams have been working on it for decades. Billions of dollars/Euros/Pounds must have been poured into it. Still, their devices and algorithms get blown out of the water by an off-the-shelf GPU plus tensorflow, pytorch, what have you. Hats off for the authors' achievement, this is no small feat and something that has been tried for years.…
interesting. where would one start reading about all this?
Re: Event-based backpropagation for exact gradients in spiking neural networks
#4This is about achieving Deep learning on Neuromorphic hardware. Large research teams have been working on it for decades. Billions of dollars/Euros/Pounds must have been poured into it. Still, their devices and algorithms get blown out of the water by an off-the-shelf GPU plus tensorflow, pytorch, what have you. Hats off for the authors' achievement, this is no small feat and something that has been tried for years.…
interesting. where would one start reading about all this?
You could start with Intel‘s Loihi Press release: https://www.intel.com/content/www/us/en/research/neuromorphi...
There you get the full dose of hype for neuromorphic computing, but without any critical reflection (naturally, since it’s a press release advertising a product).
Unfortunately I am not aware of literature that provides critical review of neuromorphic computing. You have to read between the lines of the research papers to find out that the field has failed to live up to the promise of lower-energy deep learning (which was a misguided promise from the outset, IMHO).
Re: Event-based backpropagation for exact gradients in spiking neural networks
#5Earlier quoted context omitted.
interesting. where would one start reading about all this?
You could start with Intel‘s Loihi Press release: https://www.intel.com/content/www/us/en/research/neuromorphi... There you get the full dose of hype for neuromorphic computing, but without any critical reflection (naturally, since it’s a press release advertising a product). Unfortunately I am not aware of literature that provides critical review of neuromorphic computing. You have to read between the lines of the r…
Does that include rain.ai ?
Re: Event-based backpropagation for exact gradients in spiking neural networks
#6Earlier quoted context omitted.
interesting. where would one start reading about all this?
You could start with Intel‘s Loihi Press release: https://www.intel.com/content/www/us/en/research/neuromorphi... There you get the full dose of hype for neuromorphic computing, but without any critical reflection (naturally, since it’s a press release advertising a product). Unfortunately I am not aware of literature that provides critical review of neuromorphic computing. You have to read between the lines of the r…
Could you elaborate on why you think low energy deep learning was a misguided promise for SNNs? Just came across them for the first time last week and the low energy promise seemed like their most interesting aspect!
Re: Event-based backpropagation for exact gradients in spiking neural networks
#7Earlier quoted context omitted.
You could start with Intel‘s Loihi Press release: https://www.intel.com/content/www/us/en/research/neuromorphi... There you get the full dose of hype for neuromorphic computing, but without any critical reflection (naturally, since it’s a press release advertising a product). Unfortunately I am not aware of literature that provides critical review of neuromorphic computing. You have to read between the lines of the r…
Does that include rain.ai ?
I don’t know too much about their technology and the website isn’t giving away too much detail. It doesn’t look like they are using spiking networks, so no event-based neuromorphic tech, but perhaps good old linear algebra/ANN ML. They’re using analog computation which is attractive power-wise, but in the past has always suffered from variability due to device mismatch. Unless they have some really revolutionary process or algorithm that magically makes the downsides of mismatch disappear, they’ll have a hard time going beyond what has been tried in analog computing before (and which had its heyday in the 70s).
Re: Event-based backpropagation for exact gradients in spiking neural networks
#8Earlier quoted context omitted.
You could start with Intel‘s Loihi Press release: https://www.intel.com/content/www/us/en/research/neuromorphi... There you get the full dose of hype for neuromorphic computing, but without any critical reflection (naturally, since it’s a press release advertising a product). Unfortunately I am not aware of literature that provides critical review of neuromorphic computing. You have to read between the lines of the r…
Does that include rain.ai ?
Looks like a different approach. Intel's chip is based on digital circuits. They try an analog approach.
Re: Event-based backpropagation for exact gradients in spiking neural networks
#9Earlier quoted context omitted.
You could start with Intel‘s Loihi Press release: https://www.intel.com/content/www/us/en/research/neuromorphi... There you get the full dose of hype for neuromorphic computing, but without any critical reflection (naturally, since it’s a press release advertising a product). Unfortunately I am not aware of literature that provides critical review of neuromorphic computing. You have to read between the lines of the r…
Could you elaborate on why you think low energy deep learning was a misguided promise for SNNs? Just came across them for the first time last week and the low energy promise seemed like their most interesting aspect!
Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together.
Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs).
Generally, spiking networks will likely have an edge when the signals they are processing are events in time. For example, when processing signal streams from event based sensors, like silicon retinas. There’s also evidence that event-based control has advantages over their periodically-sampling equivalents.
Re: Event-based backpropagation for exact gradients in spiking neural networks
#10One of the authors here, happy to answer any questions you might have.