Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Event-based backpropagation for exact gradients in spiking neural networks

arxiv.org

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Event-based backpropagation for exact gradients in spiking neural networks

#31

Post by orbifold »

ALLTaken wrote: post #25
orbifold wrote: post #10

One of the authors here, happy to answer any questions you might have.

I'm kind of an amateur, but incredibly curious, willing to learn and do hard work. I have many ideas and questions regarding your paper: - How do you adjust weights between different spikes? - Do you use or implement a kind of wavelet for wave-propagation, in example for spike interferences? - What neuromorphic hardware can I buy to run your code/ the SNN? =)

- Weights are adjusted at the end of the backward in time (adjoint) integration, according to the weight gradients accumulated at pre-spike times.

- We only consider one kind of model system in this paper but this method would work for any kind of hybrid dynamical system, so also other physical substrates (a lot of exciting work to do there).

- We used to sell a neuromorphic hardware system Spikey for ~3000 Euro (basically at cost), we've recently completed a similar project, we also provide access to remote users via the ebrains collaboratory (https://ebrains.eu/service/collaboratory/). There are a number of commercial offers in the works (SynSense, Inatera). You can also buy SpiNNaker boards or access them via ebrains. Loihi and TrueNorth either don't sell or are pretty expensive, but they have "research agreements" in place.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#32

Post by datameta »

periheli0n wrote: post #29
FrereKhan wrote: post #26

Earlier quoted context omitted.

It's not quite correct to say this is only for achieving deep learning. Gradient-based parameter optimisation is still a useful tool, even for small shallow networks that would be ideal for event-based signal processing. Even for small-network tasks, training spiking networks has been non-trivial. This paper provides a way to get exact gradients, implying probably faster optimisation than using surrogate gradients or…

You are totally right. The algorithm itself is a potential game-changer. I guess I was carried away by the pitch in the abstract that starts off with deep learning. Personally I think that way too many resources were wasted on trying to make better deep networks with spikes. In my opinion it is much more promising to apply spiking networks on problems that are inherently event-based. Having a functional backpropagati…

Based on reading just the abstract so far, it seems to me the event-based application of this algorithm makes absolute sense. Temporal importance can be effectively characterized in memristors. At the risk of making a comparison similar to that of Andrew Ng's a decade ago, I think this approach paired with something like a ReRAM crossbar is quite an effective rough analogue to the voltage potentials across a group of neurons in the brain.

I applaud this team's efforts. A real breakthrough.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#33

Post by orbifold »

FrereKhan wrote: post #28
orbifold wrote: post #18

Earlier quoted context omitted.

I agree with these points, however the main advantage of the method presented in the paper is precisely that both the forward propagation and backward propagation can be seen as being performed by a network operating on temporally sparse events. We absolutely had event-based sensors and control as a motivation in mind. The fact that you can write down the connectivity of the neurons in terms of a weight matrix, does…

Ultra-low-power neuromorphic processors such as DynapSE[1] have been cross-bar free for several years now, making them a perfect fit for sparse networks (both weight- and activity-sparsity). [1] https://arxiv.org/abs/1708.04198

Yes, that would be another example.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#34

Post by wombat23 »

ALLTaken wrote: post #25
orbifold wrote: post #10

One of the authors here, happy to answer any questions you might have.

I'm kind of an amateur, but incredibly curious, willing to learn and do hard work. I have many ideas and questions regarding your paper: - How do you adjust weights between different spikes? - Do you use or implement a kind of wavelet for wave-propagation, in example for spike interferences? - What neuromorphic hardware can I buy to run your code/ the SNN? =)

> What neuromorphic hardware can I buy to run your code/ the SNN?

Current neuromorphic hardware is not easily accesible, but you can simulate spiking neural networks. Check out, e.g. https://brian2.readthedocs.io/en/stable/ or Nengo.ai

Re: Event-based backpropagation for exact gradients in spiking neural networks

#35

Post by drdeca »

I'm unclear on what \tau_{syn} and \tau_{mem} mean. I assume that syn stands for synapse, and mem stands for memory, but I'm not sure what the \tau is about? Time? I imagine that this would be clear to someone in the field, who would be the target audience for the paper, so this isn't really a criticism.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#36

Post by jegp »

drdeca wrote: post #35

I'm unclear on what \tau_{syn} and \tau_{mem} mean. I assume that syn stands for synapse, and mem stands for memory, but I'm not sure what the \tau is about? Time? I imagine that this would be clear to someone in the field, who would be the target audience for the paper, so this isn't really a criticism.

\tau is a common symbol for time constants which, briefly put, determines how fast something decays over time. Syn for synaptic current and mem for membrane voltage. We've written some documentation around our neuron equations in Python that explains this: https://norse.github.io/norse/auto_api/norse.torch.functiona...

See also our tutorial on neuron parameter optimization to understand how it's useful for machine learning: https://github.com/norse/notebooks#level-intermediate

There's also a great book on the topic by Gerstner available online: https://neuronaldynamics.epfl.ch/

Disclaimer: I'm a co-author of the library Norse

Regarding the target audience, it's actually not entirely clear to me. This lies in the intersection between computational neuroscience and deep learning, which isn't a huge set of people. So, I think you're question is highly relevant and we (as researchers) have a lot of work in front of us to explain why this is interesting and important.

Re: Event-based backpropagation for exact gradients in spiking neural networks

#37

Post by drdeca »

jegp wrote: post #36
drdeca wrote: post #35

I'm unclear on what \tau_{syn} and \tau_{mem} mean. I assume that syn stands for synapse, and mem stands for memory, but I'm not sure what the \tau is about? Time? I imagine that this would be clear to someone in the field, who would be the target audience for the paper, so this isn't really a criticism.

\tau is a common symbol for time constants which, briefly put, determines how fast something decays over time. Syn for synaptic current and mem for membrane voltage. We've written some documentation around our neuron equations in Python that explains this: https://norse.github.io/norse/auto_api/norse.torch.functiona... See also our tutorial on neuron parameter optimization to understand how it's useful for machine le…

Thanks! I will take a look at those

Re: Event-based backpropagation for exact gradients in spiking neural networks

#38

Post by FrereKhan »

periheli0n wrote: post #30
FrereKhan wrote: post #27

Earlier quoted context omitted.

If you bring activation sparsity into the mix, the advantage of SNN processors over GPUs/TPUs becomes more clear. Loss-gradient-based optimisation approaches are great because they give you a tool to include e.g. sparsity regularisation into the loss. Encouraging sparse activity makes simple linear algebra a poor fit for network activation, and SNN processors a much better fit.

But is sparse activation sufficient to motivate the use of SNNs? In my opinion one needs a temporal component as well. Sparse activations that don't also have a time component (i.e. are sparse in space and time) can be very well implemented without events. Granted, SNN processors can handle sparse activations better than matrix accelerators. But then again, SNN accelerators might carry lots of SNN overhead that is no…

I agree. The use case needs to justify having state, otherwise the ideal architecture is something like NullHop. Temporal signal processing / vision processing tasks are ideal for SNNs, especially if the inputs can also be sparse.
Post reply on HN