New neural network architecture inspired by neural system of a worm
quantamagazine.org
New neural network architecture inspired by neural system of a worm
1–10 of 73 posts
Re: New neural network architecture inspired by neural system of a worm
#2As far as I can tell, they analytically solved the style of ODE used in biologically-motivated neural networks (usually spiking, but not in this case) and then trained a network built from those to do stuff.
Re: New neural network architecture inspired by neural system of a worm
#3It makes a good headline, but reading over the paper (https://www.nature.com/articles/s42256-022-00556-7.pdf) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net.
More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud feels like daring history to make a fool of me.
My view is that biology just happened to evolve how it did, so there’s no point in copying it; it worked because it worked. If we have to train networks from scratch, then we have to find our own solutions, which will necessarily be different than nature’s. I find analogies useful; dividing a model into short term memory vs long term memory, for example. But it’s best not to take it too seriously, like we’re somehow cloning a brain.
Not to mention that ML nets still don’t control their own loss functions, so we’re a poor shadow of nature. ML circa 2023 is still in the intelligent design phase, since we have to very intelligently design our networks. I await the day that ML networks can say “Ok, add more parameters here” or “Use this activation instead” (or learn an activation altogether — why isn’t that a thing?).
Re: New neural network architecture inspired by neural system of a worm
#4Although the article is recent the paper from the article has been available on preprint/arxiv since June 2021[1], implementations for pytorch & tensorflow are also available[2] for those interested.
[1]: https://arxiv.org/abs/2106.13898 [2]: https://github.com/raminmh/CfC
Re: New neural network architecture inspired by neural system of a worm
#5It makes a good headline, but reading over the paper ( https://www.nature.com/articles/s42256-022-00556-7.pdf ) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud fee…
Learned activation functions do seem to be a thing(https://arxiv.org/abs/1906.09529)
Re: New neural network architecture inspired by neural system of a worm
#6It makes a good headline, but reading over the paper ( https://www.nature.com/articles/s42256-022-00556-7.pdf ) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud fee…
The open worm project is the product of microscopically mapping the neural network (literally the biological network of neurons) in a nematode. How isn’t this biologically inspired? If I’m reading it correctly, the equations that you’re misinterpreting are the neuron models that make each node in the map. I would guess that part of the inspiration for using the word “liquid” comes from the origins of the project in which they were modeling the ion channels in the synapses.
They’ve been training these artificial nematodes to swim for years. The original project was fascinating (in a useless way): you could put the model of the worm in a physics engine and it would behave like the real-life nematode. Without any programming! It was just an emergent behavior of the mapped-out neuron models (connected to muscle models). It makes sense that they’ve isolated the useful part of the network to train it for other behaviors.
I used to follow this project, and I thought it had lost steam. Glad to see Ramin is still hard at work.
Re: New neural network architecture inspired by neural system of a worm
#7It makes a good headline, but reading over the paper ( https://www.nature.com/articles/s42256-022-00556-7.pdf ) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud fee…
Learnable activation functions are a thing famously Swish[0] is is a trainable SiLU which was found through symbolic search/optimization [1], but as it turns out that doesn't magically make make neural networks orders better.
[0]: https://en.m.wikipedia.org/wiki/Swish_function [1]: https://arxiv.org/abs/1710.05941
Re: New neural network architecture inspired by neural system of a worm
#8It makes a good headline, but reading over the paper ( https://www.nature.com/articles/s42256-022-00556-7.pdf ) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud fee…
> I’m skeptical that biological systems will ever serve as a basis for ML nets in practice
There is no fundamental difference between information processing systems implemented in silico vs in vivo, except architecture. Architecture is what constrains the manifold of internal representations: this is called "inductive bias" in the field of machine learning. The math (technically, the non-equilibrium statistical physics crossed with information theory) is fundamentally the same.
Everything at the functionalist level follows from architecture; what enables these functions is the universal principles of information processing per se. "It worked because it worked" because there is no other way for it to work given the initial conditions of our neighborhood in the universe. I'm not saying "Everything ends up looking like a brain". Rather, I am saying "The brain, attendant nervous and sensory systems, etc. vs neural networks implemented as nonlinear functions are running the same instructions on different hardware, thus resulting in different algorithms."
The way I like to put it is: trust Nature's engineers, they've been at it much longer than any of us have.
Re: New neural network architecture inspired by neural system of a worm
#9It makes a good headline, but reading over the paper ( https://www.nature.com/articles/s42256-022-00556-7.pdf ) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud fee…
The open worm project is the product of microscopically mapping the neural network (literally the biological network of neurons) in a nematode. How isn’t this biologically inspired? If I’m reading it correctly, the equations that you’re misinterpreting are the neuron models that make each node in the map. I would guess that part of the inspiration for using the word “liquid” comes from the origins of the project in w…
Interesting. Is there a way to run it?
One of the challenges with work like this is that you have to figure out how to get output from it. What would the output be?
As far as my objection, it seems like an optimization, not an architecture inspired by the worm. I.e. “inspired by” makes it sound like this particular optimization was derived from studying the worm’s neural networks and translating it into code, when it was the other way around. But it would be fascinating if that wasn’t the case.
Re: New neural network architecture inspired by neural system of a worm
#10Is there any reason to believe that biologically inspired architectures should yield better performance ? Brain are biological systems which have been trained through evolutionary processes. Neural Networks are algorithmic/linear algebra models trained through statistical methods
One might argue that CNN are biologically inspired, but it's more likely that the reason they work is because they respects input symmetries