Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event-based backpropagation for exact gradients in spiking neural networks

arxiv.org

Post reply on HN
Print view View original

11–20 of 38 posts

Re: Event-based backpropagation for exact gradients in spiking neural networks

#11

Post by clara732 »

mpfundstein wrote: post #3
periheli0n wrote: post #2

This is about achieving Deep learning on Neuromorphic hardware. Large research teams have been working on it for decades. Billions of dollars/Euros/Pounds must have been poured into it. Still, their devices and algorithms get blown out of the water by an off-the-shelf GPU plus tensorflow, pytorch, what have you. Hats off for the authors' achievement, this is no small feat and something that has been tried for years.…

interesting. where would one start reading about all this?

Here is a paper from the same group which includes actual results of an algorithm running on the neuromorphic chip: https://arxiv.org/abs/1912.11443

Re: Event-based backpropagation for exact gradients in spiking neural networks

#12

Post by haffi112 »

orbifold wrote: post #10

One of the authors here, happy to answer any questions you might have.

In table 3 you compare your result with other publications. Reference 44 has a larger accuracy than you with fewer hidden neurons. What is the difference between your method and theirs?

Also, will you release your method as code?

Re: Event-based backpropagation for exact gradients in spiking neural networks

#13

Post by orbifold »

haffi112 wrote: post #12
orbifold wrote: post #10

One of the authors here, happy to answer any questions you might have.

In table 3 you compare your result with other publications. Reference 44 has a larger accuracy than you with fewer hidden neurons. What is the difference between your method and theirs? Also, will you release your method as code?

That number is what they reported in their publication, but it turns out they actually used both recurrent neurons (compared to feedforward as they state) and 512 instead of 100 neurons (see here https://zenkelab.org/publications/errata_zenke_vogels_2021/). We will adjust those numbers in the final publication.

My aim is to release the method as part of Norse https://github.com/norse/norse. There is some subtlety involved in implementing it for a given integration scheme, though. The event based simulator underlying the paper will also be released in due time.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#14

Post by throwaway245234 »

haffi112 wrote: post #12
orbifold wrote: post #10

One of the authors here, happy to answer any questions you might have.

In table 3 you compare your result with other publications. Reference 44 has a larger accuracy than you with fewer hidden neurons. What is the difference between your method and theirs? Also, will you release your method as code?

Other author here, as was pointed out this was a typo in the cited publication. We will definitely be releasing the source code in due time.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#16

Post by marmaduke »

mpfundstein wrote: post #3
periheli0n wrote: post #2

This is about achieving Deep learning on Neuromorphic hardware. Large research teams have been working on it for decades. Billions of dollars/Euros/Pounds must have been poured into it. Still, their devices and algorithms get blown out of the water by an off-the-shelf GPU plus tensorflow, pytorch, what have you. Hats off for the authors' achievement, this is no small feat and something that has been tried for years.…

interesting. where would one start reading about all this?

The EU Brainscales project built a wafer that runs 10k times faster than real-time,

https://electronicvisions.github.io/hbp-sp9-guidebook/pm/pm_...

Re: Event-based backpropagation for exact gradients in spiking neural networks

#17

Post by periheli0n »

nairboon wrote: post #15
orbifold wrote: post #10

One of the authors here, happy to answer any questions you might have.

Could you comment on this general SNN critique? https://news.ycombinator.com/item?id=27366059

As the author of that "general SNN critique" I'd like to add that this was not a general critique, but a specific reply to the question why I think low-energy deep learning is a misguided promise for SNNs.

Personally I think SNNs are a very exciting research field, both from a neuroscience as from a computer science angle. The work we are discussing here is deeply impressing for its rigour, and it addresses an important problem in spiking network research.

Whether spiking networks will provide lower-energy deep learning is a totally different question.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#18

Post by orbifold »

periheli0n wrote: post #9
dtjohnnyb wrote: post #6

Earlier quoted context omitted.

Could you elaborate on why you think low energy deep learning was a misguided promise for SNNs? Just came across them for the first time last week and the low energy promise seemed like their most interesting aspect!

Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together. Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs). Generally, spiking networks will likely ha…

I agree with these points, however the main advantage of the method presented in the paper is precisely that both the forward propagation and backward propagation can be seen as being performed by a network operating on temporally sparse events. We absolutely had event-based sensors and control as a motivation in mind. The fact that you can write down the connectivity of the neurons in terms of a weight matrix, does not mean that it can't be sparse. Since you are actually processing one spike at a time (potentially asynchronously), you don't need to implement any matrix multiplication. Current neuromorphic hardware achieves at least some degree of sparsity in their synaptic crossbars (BrainScales2, Spinnaker) or largely eliminates them like Loihi.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#19

Post by orbifold »

periheli0n wrote: post #17
nairboon wrote: post #15

Earlier quoted context omitted.

Could you comment on this general SNN critique? https://news.ycombinator.com/item?id=27366059

As the author of that "general SNN critique" I'd like to add that this was not a general critique, but a specific reply to the question why I think low-energy deep learning is a misguided promise for SNNs. Personally I think SNNs are a very exciting research field, both from a neuroscience as from a computer science angle. The work we are discussing here is deeply impressing for its rigour, and it addresses an import…

I actually think I largely agree with the points that you made in the comment thread above. The name of the game can't be to just map a deep neural network to a SNN, I find it far more interesting to identify more natural mechanisms by which SNNs are able to perform information processing on sparse asynchronous event-based data. Also thank you for the nice words :).

Re: Event-based backpropagation for exact gradients in spiking neural networks

#20

Post by periheli0n »

orbifold wrote: post #18
periheli0n wrote: post #9

Earlier quoted context omitted.

Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together. Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs). Generally, spiking networks will likely ha…

I agree with these points, however the main advantage of the method presented in the paper is precisely that both the forward propagation and backward propagation can be seen as being performed by a network operating on temporally sparse events. We absolutely had event-based sensors and control as a motivation in mind. The fact that you can write down the connectivity of the neurons in terms of a weight matrix, does…

Yes, the algorithm you proposed is impressive and has the potential to become a game-changer.

However, I think the MNIST and the Ying/Yang dataset, using latency-coding, are not the ideal example to demonstrate its performance.

These datasets are useful to demonstrate nonlinear classification, and it's certainly great to see that the spiking network performs competitively. However, the transformation into a latency code costs time, in terms of computation, and also in terms of representation, before even one item is classified. Perceptron-based ANNs with continuous outputs don't require this step and will always have an edge over spiking networks in such scenarios.

I think what the field is really lacking is an ML problem that can leverage spiking networks directly, that does not require costly conversion of data into a representation that is suitable for spiking networks.

Post reply on HN