[deleted]
New neural network architecture inspired by neural system of a worm
31–40 of 73 posts
Re: New neural network architecture inspired by neural system of a worm
#32so liquid nets don't need inputs of a fixed length?
Re: New neural network architecture inspired by neural system of a worm
#33It makes a good headline, but reading over the paper ( https://www.nature.com/articles/s42256-022-00556-7.pdf ) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud fee…
"I’m skeptical that biological systems will ever serve as a basis for ML nets in practice" First of all, ML engineers need to stop being so brainphiliacs, caring only about the 'neural networks' of the brain or brain-like systems. Lacrymaria olor has more intelligence, in terms of adapting to exploring/exploiting a given environment, than all our artificial neural networks combined and it has no neurons because it is…
[1] linked above is an absolute powerhouse of a lecture by Michael Levin. Wow.
Re: New neural network architecture inspired by neural system of a worm
#34It makes a good headline, but reading over the paper ( https://www.nature.com/articles/s42256-022-00556-7.pdf ) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud fee…
> so there’s no point in copying it
Not sure about that, a lot of solutions in nature are honed by billions of years of evolution, sometimes creating feats even more impressive than we can do currently. There is an entire field about copying biology to solve our problems:
Re: New neural network architecture inspired by neural system of a worm
#35It makes a good headline, but reading over the paper ( https://www.nature.com/articles/s42256-022-00556-7.pdf ) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud fee…
> It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net.
You say that like it isn’t a big deal. Finding an analytical solution to optimising the parameters of a non-linear equation is remarkable.
Re: New neural network architecture inspired by neural system of a worm
#36Imo, the next step in ML is unleashing the electron a bit. Right now we keep probabilistic electrons on leash in transistors, so they behave deterministically. This despotic method has taken us far, but without giving electrons some freedom back, we won't advance further.
Re: New neural network architecture inspired by neural system of a worm
#37It makes a good headline, but reading over the paper ( https://www.nature.com/articles/s42256-022-00556-7.pdf ) it doesn’t seem biologically-inspired. It seems like they found a way to solve nonlinear equations in constant time via an approximation, then turned that into a neural net. More generally, I’m skeptical that biological systems will ever serve as a basis for ML nets in practice. But saying that out loud fee…
Well I tend to agree but you seem to think biology evolved in a vacuum but it evolved inside an information source and we're all the information processors: ML having to process the same information but at scale, in the grand scheme, it will probably have to ressemble a brain in some ways. Just the sources we care about (colors in a picture, faces, prices, wind direction, whatever) and the output we can understand (text, images, sound) will have to skew it towards us in the way it has to model things.
Re: New neural network architecture inspired by neural system of a worm
#38Earlier quoted context omitted.
> I’m skeptical that biological systems will ever serve as a basis for ML nets in practice There is no fundamental difference between information processing systems implemented in silico vs in vivo, except architecture. Architecture is what constrains the manifold of internal representations: this is called "inductive bias" in the field of machine learning. The math (technically, the non-equilibrium statistical physi…
> There is no fundamental difference between information processing in silicon and in vivo A neuron has dozens of neurotransmitters, while artificial neurons produce 1 output. I don't know much about neurology, but how is the information processing similar? What do you mean are running the same instructions? > there is no other way for it to work Plants exhibit learned behaviors
> how is the information processing similar?
The representational capacities are of course not the same -- the same "thoughts" cannot be expressed in both systems. But the concept of "processing over abstract representations enacted in physical dynamics within cognitive systems" is shared between all systems of this kind.
I am referring to "information processing" at the physical level, i.e., "'useful' work per energy quantum as communicated through noisy channels".
> What do you mean are running the same instructions?
The underlying physical principles of such information processing are equivalent regardless of physical implementation.
> plants exhibit learned behaviors
A good example of what I mean. The architecture is different, but the underlying dynamics is the same.
There is a convincing (to me) theory of the origins of life[1][2] that states that thermodynamics -- and, by extension, information theory -- is the appropriate level of abstraction for understanding what distinguishes living processes from inanimate ones. The theory posits that a system, well-defined by some (possibly arbitrary) boundaries, "learns" (develops channels through which "patterns" can be "recognized" and possibly interacted with) as an inevitable result of physics. Put another way, a learning system is one that represents its experiences through the cumulative wearing-in over time of channels of energy flows.
What concepts the system can possibly represent depends on in what ways the system can wear while maintaining its essential functions. What specifically the system learns is the set of concepts which collectively best communicate (physically, i.e., from the "inputs" through the "processing" functions and to the "outputs") the historical set of its experiences of its environment and of itself.
I want to note that this discussion has nothing to say on perception, only sensation and reaction: in other words, it is an exclusively materialist analysis.
Optimization theory describes its notion of learning roughly as such (considering "loss" as energy potentials), but with the same language we could also describe a human brain, or a black hole's accretion disk, or an ant colony dug deep into clay.
References:
[1] https://www.englandlab.com/uploads/7/8/0/3/7803054/2013jcpsr...
[2] https://www.quantamagazine.org/a-new-thermodynamics-theory-o...
Parallel directions of research:
Re: New neural network architecture inspired by neural system of a worm
#39Earlier quoted context omitted.
> I’m skeptical that biological systems will ever serve as a basis for ML nets in practice There is no fundamental difference between information processing systems implemented in silico vs in vivo, except architecture. Architecture is what constrains the manifold of internal representations: this is called "inductive bias" in the field of machine learning. The math (technically, the non-equilibrium statistical physi…
> There is no fundamental difference between information processing in silicon and in vivo A neuron has dozens of neurotransmitters, while artificial neurons produce 1 output. I don't know much about neurology, but how is the information processing similar? What do you mean are running the same instructions? > there is no other way for it to work Plants exhibit learned behaviors
> A neuron has dozens of neurotransmitters, while artificial neurons produce 1 output. I don't know much about neurology, but how is the information processing similar? What do you mean are running the same instructions?
ANNs are general function approximations. You can get the same behaviour from a complex network of simple neurons that you get from a single more complex neuron.
Re: New neural network architecture inspired by neural system of a worm
#40The old neuroscience saying goes like this: "Human brain have billions of neurons and so it is too complex to understand, that's why neuroscience study simpler organisms. Flatworm's brain have 52 neurons. We have no idea how it works". Did finally something changed in this regard?
Yes. The C. Elegans brain (~300 neurons) was the first organism to be completely mapped to a connectome (the map of all connections). The first complete connectome of any centralized brain, the fruit fly, is about to be completed by the Flywire project ( https://home.flywire.ai/ ) ~100,000 neurons and ~70,000,000 synapses. We have just a little idea how it works ;)
A bit of extrapolation might suggested we could map out the connectome of a human brain in 40-50 years. Not that I’d suggest a linear extrapolation from two data points…