Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event-based backpropagation for exact gradients in spiking neural networks

arxiv.org

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Event-based backpropagation for exact gradients in spiking neural networks

#21

Post by nairboon »

periheli0n wrote: post #17
nairboon wrote: post #15

Earlier quoted context omitted.

Could you comment on this general SNN critique? https://news.ycombinator.com/item?id=27366059

As the author of that "general SNN critique" I'd like to add that this was not a general critique, but a specific reply to the question why I think low-energy deep learning is a misguided promise for SNNs. Personally I think SNNs are a very exciting research field, both from a neuroscience as from a computer science angle. The work we are discussing here is deeply impressing for its rigour, and it addresses an import…

Maybe "critique" was the wrong choice of word. Your point is now much more clear to me, thank you for your insights.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#22

Post by krishnadevi »

nairboon wrote: post #21
periheli0n wrote: post #17

Earlier quoted context omitted.

As the author of that "general SNN critique" I'd like to add that this was not a general critique, but a specific reply to the question why I think low-energy deep learning is a misguided promise for SNNs. Personally I think SNNs are a very exciting research field, both from a neuroscience as from a computer science angle. The work we are discussing here is deeply impressing for its rigour, and it addresses an import…

Maybe "critique" was the wrong choice of word. Your point is now much more clear to me, thank you for your insights.

Absolutely right

Re: Event-based backpropagation for exact gradients in spiking neural networks

#23

Post by orbifold »

periheli0n wrote: post #20
orbifold wrote: post #18

Earlier quoted context omitted.

I agree with these points, however the main advantage of the method presented in the paper is precisely that both the forward propagation and backward propagation can be seen as being performed by a network operating on temporally sparse events. We absolutely had event-based sensors and control as a motivation in mind. The fact that you can write down the connectivity of the neurons in terms of a weight matrix, does…

Yes, the algorithm you proposed is impressive and has the potential to become a game-changer. However, I think the MNIST and the Ying/Yang dataset, using latency-coding, are not the ideal example to demonstrate its performance. These datasets are useful to demonstrate nonlinear classification, and it's certainly great to see that the spiking network performs competitively. However, the transformation into a latency c…

I agree that the choice of task is not ideal. It is something that I struggled quite a bit with, since coming up with a good task can be a lot of work. Unfortunately even some of the "neuromorphic" datasets that are in use can be solved by massive temporal averaging or result in reduced performance of the network relative to "analog" temporal input (e.g. on Google Speech Commands). I'm collaborating with a group that is interested in event-based vision and control, so hopefully this will result in more practical/impressive demonstrations in the future.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#24

Post by PaulHoule »

periheli0n wrote: post #20
orbifold wrote: post #18

Earlier quoted context omitted.

I agree with these points, however the main advantage of the method presented in the paper is precisely that both the forward propagation and backward propagation can be seen as being performed by a network operating on temporally sparse events. We absolutely had event-based sensors and control as a motivation in mind. The fact that you can write down the connectivity of the neurons in terms of a weight matrix, does…

Yes, the algorithm you proposed is impressive and has the potential to become a game-changer. However, I think the MNIST and the Ying/Yang dataset, using latency-coding, are not the ideal example to demonstrate its performance. These datasets are useful to demonstrate nonlinear classification, and it's certainly great to see that the spiking network performs competitively. However, the transformation into a latency c…

I have always wondered if results against the MNIST digits are generic. One might think it would work if you put in some other digits such as 一, 二, 三, 四 would they cluster the same with tSNE?

Re: Event-based backpropagation for exact gradients in spiking neural networks

#25

Post by ALLTaken »

orbifold wrote: post #10

One of the authors here, happy to answer any questions you might have.

I'm kind of an amateur, but incredibly curious, willing to learn and do hard work.

I have many ideas and questions regarding your paper:

- How do you adjust weights between different spikes?

- Do you use or implement a kind of wavelet for wave-propagation, in example for spike interferences?

- What neuromorphic hardware can I buy to run your code/ the SNN?

=)

Re: Event-based backpropagation for exact gradients in spiking neural networks

#26

Post by FrereKhan »

periheli0n wrote: post #2

This is about achieving Deep learning on Neuromorphic hardware. Large research teams have been working on it for decades. Billions of dollars/Euros/Pounds must have been poured into it. Still, their devices and algorithms get blown out of the water by an off-the-shelf GPU plus tensorflow, pytorch, what have you. Hats off for the authors' achievement, this is no small feat and something that has been tried for years.…

It's not quite correct to say this is only for achieving deep learning. Gradient-based parameter optimisation is still a useful tool, even for small shallow networks that would be ideal for event-based signal processing.

Even for small-network tasks, training spiking networks has been non-trivial. This paper provides a way to get exact gradients, implying probably faster optimisation than using surrogate gradients or other approximation methods for SNNs.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#27

Post by FrereKhan »

periheli0n wrote: post #9
dtjohnnyb wrote: post #6

Earlier quoted context omitted.

Could you elaborate on why you think low energy deep learning was a misguided promise for SNNs? Just came across them for the first time last week and the low energy promise seemed like their most interesting aspect!

Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together. Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs). Generally, spiking networks will likely ha…

If you bring activation sparsity into the mix, the advantage of SNN processors over GPUs/TPUs becomes more clear. Loss-gradient-based optimisation approaches are great because they give you a tool to include e.g. sparsity regularisation into the loss. Encouraging sparse activity makes simple linear algebra a poor fit for network activation, and SNN processors a much better fit.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#28

Post by FrereKhan »

orbifold wrote: post #18
periheli0n wrote: post #9

Earlier quoted context omitted.

Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together. Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs). Generally, spiking networks will likely ha…

I agree with these points, however the main advantage of the method presented in the paper is precisely that both the forward propagation and backward propagation can be seen as being performed by a network operating on temporally sparse events. We absolutely had event-based sensors and control as a motivation in mind. The fact that you can write down the connectivity of the neurons in terms of a weight matrix, does…

Ultra-low-power neuromorphic processors such as DynapSE[1] have been cross-bar free for several years now, making them a perfect fit for sparse networks (both weight- and activity-sparsity). [1] https://arxiv.org/abs/1708.04198

Re: Event-based backpropagation for exact gradients in spiking neural networks

#29

Post by periheli0n »

FrereKhan wrote: post #26
periheli0n wrote: post #2

This is about achieving Deep learning on Neuromorphic hardware. Large research teams have been working on it for decades. Billions of dollars/Euros/Pounds must have been poured into it. Still, their devices and algorithms get blown out of the water by an off-the-shelf GPU plus tensorflow, pytorch, what have you. Hats off for the authors' achievement, this is no small feat and something that has been tried for years.…

It's not quite correct to say this is only for achieving deep learning. Gradient-based parameter optimisation is still a useful tool, even for small shallow networks that would be ideal for event-based signal processing. Even for small-network tasks, training spiking networks has been non-trivial. This paper provides a way to get exact gradients, implying probably faster optimisation than using surrogate gradients or…

You are totally right. The algorithm itself is a potential game-changer. I guess I was carried away by the pitch in the abstract that starts off with deep learning.

Personally I think that way too many resources were wasted on trying to make better deep networks with spikes. In my opinion it is much more promising to apply spiking networks on problems that are inherently event-based.

Having a functional backpropagation algorithm such as the one provided can help with that, obviously.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#30

Post by periheli0n »

FrereKhan wrote: post #27
periheli0n wrote: post #9

Earlier quoted context omitted.

Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together. Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs). Generally, spiking networks will likely ha…

If you bring activation sparsity into the mix, the advantage of SNN processors over GPUs/TPUs becomes more clear. Loss-gradient-based optimisation approaches are great because they give you a tool to include e.g. sparsity regularisation into the loss. Encouraging sparse activity makes simple linear algebra a poor fit for network activation, and SNN processors a much better fit.

But is sparse activation sufficient to motivate the use of SNNs? In my opinion one needs a temporal component as well.

Sparse activations that don't also have a time component (i.e. are sparse in space and time) can be very well implemented without events.

Granted, SNN processors can handle sparse activations better than matrix accelerators. But then again, SNN accelerators might carry lots of SNN overhead that is not required for sparse activations alone.

Edit: A good example for a non-spiking sparse activation accelerator is the NullHop architecture [1].

[1] https://ieeexplore.ieee.org/abstract/document/8421093

Post reply on HN