Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computational Power Found in the Arms of Neurons

quantamagazine.org

Post reply on HN
Print view View original

71–78 of 78 posts

Re: Computational Power Found in the Arms of Neurons

#71

Post by AstralStorm »

RocketSyntax wrote: post #47

Ah, it's spiking edges as opposed to spiking node activations. I like this because it applies to 1 of many connections uniquely, rather than dividing the entire graph by spiking neurons. Exponentially more edges than nodes in a dense net. Connections are always more important than entities. ```The dendrites generated local spikes, had their own nonlinear input-output curves and had their own activation thresholds, di…

Change the simulation unit from neurons to dendrites. Miss different kinds of gate and nonlinear activations.

Which is what's being done lately, but then we know next to nothing about the topology of real neural nets nor electrochemical communication inside a neuron.

Re: Computational Power Found in the Arms of Neurons

#72

Post by deckar01 »

gambler wrote: post #57
deckar01 wrote: post #31

> It may also prompt some computer scientists to reappraise strategies for artificial neural networks, which have traditionally been built based on a view of neurons as simple, unintelligent switches. It has been suggested that neural networks have diverged from biological principles to the point where biological research does not provide any useful improvement to current machine learning techniques. Back propagation…

>Biology seems much more brute force in comparison. A singly honeybee with its billion synapses can adapt to a variety of situations and environments, learn on-the-fly, perform complex sequences of tasks, cooperate and communicate with other bees. All of these capabilities are emergent and packed in its tiny head. State of the art artificial neural networks (ranging beyond billions of parameters now) only do the thin…

The majority of behavior in organisms like bees is instinctual, not learned in their lifetime. That training required millions of billions of trials over the course of hundreds of millions of years.

Re: Computational Power Found in the Arms of Neurons

#74

Post by neural_thing »

neural_thing wrote: post #73

I have written a short, clear, and dense book about this exact subject. It's free! Read it here: http://www.corticalcircuitry.com/

A preprint that came out after I released the book that I would have mentioned:

https://www.biorxiv.org/content/10.1101/613141v1.full

They model a single cortical neuron as a deep neural network with 7 hidden layers consisting of 128 hidden units each.

Biological neurons are A LOT more powerful than "neurons" in neural networks. If you hear a claim about computer-brain parity being close - the people making it almost certainly don't understand the power of cortical neurons.

Re: Computational Power Found in the Arms of Neurons

#75

Post by gambler »

deckar01 wrote: post #72
gambler wrote: post #57

Earlier quoted context omitted.

>Biology seems much more brute force in comparison. A singly honeybee with its billion synapses can adapt to a variety of situations and environments, learn on-the-fly, perform complex sequences of tasks, cooperate and communicate with other bees. All of these capabilities are emergent and packed in its tiny head. State of the art artificial neural networks (ranging beyond billions of parameters now) only do the thin…

The majority of behavior in organisms like bees is instinctual, not learned in their lifetime. That training required millions of billions of trials over the course of hundreds of millions of years.

>The majority of behavior in organisms like bees is instinctual, not learned in their lifetime.

It doesn't matter whether their behavior is considered "instinctual". What matters is that they can quickly adapt their behavior to entirely novel scenarios:

https://science.sciencemag.org/content/355/6327/833

Re: Computational Power Found in the Arms of Neurons

#76

Post by K0SM0S »

ASlave2Gravity wrote: post #67
K0SM0S wrote: post #44

Earlier quoted context omitted.

You could argue that "man-made machines", "biological machines" (all of life, animal, vegetal and fungus) and "simple matter machines" (like stars are engines or telluric planets are combustion heaters), even the "surface natural ecosystem of Earth", are all different material implementations of machines: organized systems. Carbon-based, copper-based, hydrogen-based... you might map the whole periodic table of elemen…

> Because perception is so centered on thought, we tend to overappreciate the importance of thinking in our lives, in our behaviors and even values Very true. Thought can become our master easily. As a programmer, I often find myself struggling to get out of 'logic' mode or 'thinking' mode. > I'll let you ponder the discrepancy from an absolutely (perhaps) low-level logical machine to the possibility of a chaotic non…

> I think the ideas of emergent behaviours are a saving grace for the incessant desire of ours to reduce everything to its atomic (in the sense of an atomic operation, not an atom) components.

Very well said! Humans think a lot in terms of dichotomies (here the micro/macro, or whichever scale you want to consider), but I've heard biologists and physicists explaining that the closest to fundamental functions is closer to e (natural log), sinus (and hyperbolic stuff), the notion of "binary polarity" is very specific and quite limited — look at the Standard Model, it's clearly more complicated.

I personally think that's how the dimension of time emerges in our perception (any perfect observer): periodicity in all phenomena (but with e.g. Fourier it gets extremely complex at emergent thresholds), the arrow of time (non-periodicity is basically high entropy, 'heat death'/homogeneity).

Cue any such human-driven dichotomy (I think this one is relatively easy: the deepest processing system is probably close to "emotions", and these at the lowest level work in terms of "rather good" and "rather bad" (different sub-regions of the brain) + some "general integration" (third region). And what do you know, we tend to be heavily polarized in general, it's like most people feel reassured (familiarity) when things are explained in terms of black and white, left and right, ones and zeros.

(Just my thoughts on it.)

> My wider concern is that we are currently obsessed with analytical thought and philosophy and that the continental philosophy has become a 2nd class citizen. I think this has happened becuase analytical thought lends itself to being algorithmized, whereas continental philosophy does not.

Strongly agreed. I think we're witnessing a kind of "revival" through many fields (some spiritual, some historical). Things are moving. I think dumb things like numbers also influence people, it's been ~20 years since the turn of the millenium and that's enough for 1 generation to weigh in and the others to accept change because "oh new era, obviously, new number!" — this plays at the subconscious level of very crude processing, me thinks.

Nice tangent, hopefully not too hyperbolic. ;-)

Re: Computational Power Found in the Arms of Neurons

#77

Post by bnegreve »

AstralStorm wrote: post #70
bnegreve wrote: post #65

Earlier quoted context omitted.

> The risk is that we've developed a formalism and ecosystem that works on entirely different principles [...] It's a possibility, but I sometime have the feeling that people dismiss the idea that such a simple model of the brain could be enough to explain complex behaviors because they want to believe there is more to the brain. I'm sure real neurons are very complex and difficult to model, but I also believe that t…

And why do you believe the current non-intelligent ML is sufficient? As it is, ML is running away rather fast from the integrator model by introducing explicit gating and nonlinearities in the neurons. Yet it is still a toy compared to a nematode.

You're right, I doubt that the current models are enough to capture the complexity of the brain.

But models based on incredibly simple neurons can already produce quite complex behaviors. They show how many simple computing units interacting with each others can lead to things like vision. And I do believe that this is a fundamental principle.

Maybe we should explore that idea and scale this model up, instead of rejecting it as "too simple" and hoping that the complexity of the brain will be fully explained by the discovery of some quantum effect in neurons.

> As it is, ML is running away rather fast from the integrator model by introducing explicit gating and nonlinearities in the neurons.

I think the idea of a non-linear activation function has always been around. But for the rest I agree.

Re: Computational Power Found in the Arms of Neurons

#78

Post by bordercases »

api wrote: post #56

As someone who studied neurobiology more deeply than most CS people I've been talking into the wind for a long time that neurons are not point objects that can be simply modeled by equations. They are organisms with incredibly complex behaviors and internal gene regulatory structures and other information processing capabilities. Neural networks are coarse grained models of large scale brain structure. The fact that…

Amen.
Post reply on HN