Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computational Power Found in the Arms of Neurons

quantamagazine.org

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Computational Power Found in the Arms of Neurons

#51

Post by cyorir »

hinkley wrote: post #46
bonoboTP wrote: post #10

Knowing little about neuroscience outside of biology class, I've always wondered why neurons would be so simple as to just compute weighted sums and thresholds. I mean there are unicellular creatures that can do incredibly complex things, and neurons, as all cells, are ultimately the descendants of such unicellular organisms, "teamed up" into a multi-cellular creature. They have a lot of internal structure, organelle…

Around the time everyone was talking about tensegrity there was a researcher that had figured out a mechanism by which cells can sense pressure, due I think to mechanical stress on the molecules in membrane being detectable inside the cell. Proteins and lipids as proverbial nerve endings. They speculated from this that neurons rely on quantum effects, possibly using similar mechanisms, and that the surface area is in…

Surface area of the neuron depends on the type of neuron. It can be a very important factor in summation of inputs and propagation of potentials, along with shape, diameter, myelination, etc.

Re: Computational Power Found in the Arms of Neurons

#52

Post by Fishysoup »

bonoboTP wrote: post #10

Knowing little about neuroscience outside of biology class, I've always wondered why neurons would be so simple as to just compute weighted sums and thresholds. I mean there are unicellular creatures that can do incredibly complex things, and neurons, as all cells, are ultimately the descendants of such unicellular organisms, "teamed up" into a multi-cellular creature. They have a lot of internal structure, organelle…

We always knew that neurons did way more than just integrate voltage and produce spikes. The problem with testing out what those things are has largely been technological. Only until quite recently have we been able to electrically isolate dendrites and axons for recording, and slowly it's becoming tractable to image from entire neurons with high temporal resolution using voltage-sensitive fluorescent indicators. So the field of dendritic and other sub-neuronal computation arguably in its infancy.

The other things we figured out in the meantime relate to different types of neurotransmission (e.g. volume neurotransmission), active processes in axons and dendrites like vesicular trafficking, synaptic tag-and-capture (if it turns out to be true) and all kinds of weird types of plasticity. Basically a neuron's function is a lot more nuanced than the simplistic "integrate and fire" idea.

Artificial neural networks (in the deep learning sense) therefore don't really have much of anything to do with biological ones function-wise. It really is just regression with a lot of nodes and layers, and maybe some bells and whistles. That doesn't mean they're not powerful in their own right, just not comparable to brain circuits and don't do the same kind of thing nor solve the same kinds of tasks.

Re: Computational Power Found in the Arms of Neurons

#53

Post by cyorir »

whatshisface wrote: post #35

Question for ML people: would backpropagation work if some of the neurons had nonmonotonic activation functions (for example exp(-x²)), or would the gradient descent get stuck on one side or the other?

Gradient descent should have no issues with a function like exp(-x^2). Actually, softmax (softmax(x_i) = exp(x_i)/sum_j(exp(x_j))) is sometimes used as an activation function. It could make sense to modify the softmax function to use -x^2 in place of x, for some use case. However, it doesn't always make sense as a drop-in replacement for other activation functions like ReLu or Sigmoid. It really depends on your use case.

Re: Computational Power Found in the Arms of Neurons

#54

Post by radarsat1 »

deckar01 wrote: post #31

> It may also prompt some computer scientists to reappraise strategies for artificial neural networks, which have traditionally been built based on a view of neurons as simple, unintelligent switches. It has been suggested that neural networks have diverged from biological principles to the point where biological research does not provide any useful improvement to current machine learning techniques. Back propagation…

> It may also prompt some computer scientists to reappraise strategies for artificial neural networks, which have traditionally been built based on a view of neurons as simple, unintelligent switches.

I find this is a conveniently simplistic view of the current state of machine learning, which they wiggle out of by saying "traditionally built". Anyone working on ML and neural networks these days knows that modern networks are built out of building blocks -- certain standard architecture archetypes (MLP vs CNN, autoencoders, GANs, etc), resnet blocks, training regimes (layer pre-training, progressive training, etc), normalization methods, and so on. Sure, the term "neural networks" originally implied that each linear-non-linear operation pair represented a single neuron, but it's not a stretch to say that if neurons are more powerful than we thought, then each neuron represents, say, a small MLP block, or a resnet block. The biological analogy still stands. Citing ideas of ANNs from two decades ago to promise how new results in biology may change "current" thinking in ML (by citing decades-old ideas while calling them "traditional" no less!) is disingenuous.

Re: Computational Power Found in the Arms of Neurons

#55

Post by aratakareigen »

ASlave2Gravity wrote: post #33
nabla9 wrote: post #24

Earlier quoted context omitted.

Turing machine is mathematical model for computation. Deterministic Turing machine (DTM) is computationally equivalent to Non-Deterministic Turing machine (NTM) but the time complexity may not be the same. >What I'm ultimately interested in is if we, as humans, are operating in a realm that logic cannot operate in. If that makes any sense? Any mix of logic and randomness is still a machine. You must assume something…

> You must assume something spiritual if you think there exist rational action without computation behind it. I'm still uncertain i.e. spiritual. I'm finding, as I grow older, logic and reason fail to answer for my experiences. Whether that's a shortcoming in my own ability to comprehend logic and reason, or because logic and reason simply do not have all the answers, I do not know. > Randomness does not help there.…

What is the purpose of life, which you say depends on randomness?

Re: Computational Power Found in the Arms of Neurons

#56

Post by api »

As someone who studied neurobiology more deeply than most CS people I've been talking into the wind for a long time that neurons are not point objects that can be simply modeled by equations. They are organisms with incredibly complex behaviors and internal gene regulatory structures and other information processing capabilities.

Neural networks are coarse grained models of large scale brain structure. The fact that these models can be taught to do very interesting brain-like things (especially pattern recognition) demonstrates that this structure is important and fundamental, but that doesn't mean it's the whole picture of what's going on in the brain.

I say talking into the wind because in my experience most CS people tend to hand-wave away biology at the cellular level and below. There's this zealotry around us being "almost there" with AGI and the rest that blinds people to the real magnitude of the problem and how much is really happening in the brain. This likely includes a whole lot that we haven't even started to really understand.

Re: Computational Power Found in the Arms of Neurons

#57

Post by gambler »

deckar01 wrote: post #31

> It may also prompt some computer scientists to reappraise strategies for artificial neural networks, which have traditionally been built based on a view of neurons as simple, unintelligent switches. It has been suggested that neural networks have diverged from biological principles to the point where biological research does not provide any useful improvement to current machine learning techniques. Back propagation…

>Biology seems much more brute force in comparison.

A singly honeybee with its billion synapses can adapt to a variety of situations and environments, learn on-the-fly, perform complex sequences of tasks, cooperate and communicate with other bees. All of these capabilities are emergent and packed in its tiny head.

State of the art artificial neural networks (ranging beyond billions of parameters now) only do the thing they're specifically built for, only after training with bazillion specific examples and consume tons of energy while doing so.

Which one of these sounds like the brute force approach?

Re: Computational Power Found in the Arms of Neurons

#58

Post by K0SM0S »

K0SM0S wrote: post #44
ASlave2Gravity wrote: post #14

Earlier quoted context omitted.

That's a good question. I guess I'd say a Turing machine is my definition of a machine. And I suppose the implication is that if we are machines, operating on a stream of input, does that make us deterministic? EDIT: What I'm ultimately interested in is if we, as humans, are operating in a realm that logic cannot operate in. If that makes any sense? I'll probably have to define realm!

You could argue that "man-made machines", "biological machines" (all of life, animal, vegetal and fungus) and "simple matter machines" (like stars are engines or telluric planets are combustion heaters), even the "surface natural ecosystem of Earth", are all different material implementations of machines: organized systems. Carbon-based, copper-based, hydrogen-based... you might map the whole periodic table of elemen…

edit: no sure what that website is, it was a google search. Here's another source: https://phys.org/news/2014-11-filamentary-galaxies-evolve-co...

Re: Computational Power Found in the Arms of Neurons

#59

Post by cafebeen »

orbifold wrote: post #37

For those of you that want to be horrified / amused by code, check out the materials and methods of the article itself. It contains a link to the full source code used to generate the modeling figures: https://senselab.med.yale.edu/ModelDB/showmodel.cshtml?model... The simulator used is called Neuron. Code for it is written in a custom language called Hoc and models are implemented in yet another domain specific lang…

Alas, the possibility of comments like this are one reason that some researchers hesitate to release their code.

Even if there are issues, it's tremendously valuable that they shared this, so others can reproduce and build on their work. That said, most scientists would like to have better quality code in science as well, but I think the incentive structures and other demands that make it challenging...

Re: Computational Power Found in the Arms of Neurons

#60

Post by chmod775 »

dharma1 wrote: post #49
chmod775 wrote: post #29

Earlier quoted context omitted.

From the top reddit comment: > This paper is amazing. What is missing from the description above is that this is the first example of how human neurons are qualitatively different than rodent neurons (not only more computation power, but categorically different computation). > ELI5: the way the biological human neuron implements XOR is by a formerly unknown type of local response to inputs, which is low below the thr…

Could you link the Reddit comment please?

Oh. The link was changed on HN.

Here is the reddit comment: https://www.reddit.com/r/MachineLearning/comments/ejbwvb/r_s...

Post reply on HN