Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single cortical neurons as deep artificial neural networks

sciencedirect.com

Post reply on HN
Print view View original

71–80 of 135 posts

Re: Single cortical neurons as deep artificial neural networks

#71

Post by phkahler »

causi wrote: post #63
zetalyrae wrote: post #2

ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…

Indeed. Building an AI that matches human intelligence using equal or less mass than a human brain requires one or both of two things to be true: 1. The computational mass efficiency of brain tissue is very far from optimum. Considering the amount of time evolution has been improving upon it, I highly doubt that is true. 2. Most of the brain's computation is not involved in cognition. That may be true. We don't reall…

>> A dragonfly can take inputs from thousands of ommatidia and use them to track targets in space using only sixteen neurons.

Amazing. Do you have a reference for that? I'd like to read about the findings as well as how they determined that!

Re: Single cortical neurons as deep artificial neural networks

#72

Post by sdenton4 »

causi wrote: post #67
sdenton4 wrote: post #65

Earlier quoted context omitted.

/2. Most of the brain's computation is not involved in cognition. That may be true. We don't really know./ I thought this at least was fairly well understood: We do in fact use our whole brains, as anything less would be a fantastic waste of resources, which evolution would have taken care of long ago. We have numerous human-specific adaptations to deal with the relatively massive brains we're carrying around.

When you're building an AI you may not need the neurons involved with, for example, breathing. That's what I'm talking about. I'm not a neuroscientist so I don't know for sure whether all the neurons we use for muscle and organ control do double-duty to help us cogitate.

Depending on what you think about the AGI-ness of other mammals, this could be helpful to meditate upon:

https://en.wikipedia.org/wiki/Brain-to-body_mass_ratio

"In animals, it is thought that the larger the brain, the more brain weight will be available for more complex cognitive tasks. However, large animals need more neurons to represent their own bodies and control specific muscles;[clarification needed][citation needed] thus, relative rather than absolute brain size makes for a ranking of animals that better coincides with the observed complexity of animal behaviour. The relationship between brain-to-body mass ratio and complexity of behaviour is not perfect as other factors also influence intelligence, like the evolution of the recent cerebral cortex and different degrees of brain folding,[5] which increase the surface of the cortex, which is positively correlated in humans to intelligence."

Re: Single cortical neurons as deep artificial neural networks

#73

Post by caddemon »

vikramkr wrote: post #64
caddemon wrote: post #56

Earlier quoted context omitted.

It's arguable that some memories are stored via epigenetic modifications to DNA, depending on how you define "storing memories". It is very unlikely it is storing memories in the way humans typically think of memories. But I don't think "in DNA" necessarily implies "in DNA sequence".

In dna definitely implies in dna sequence. Especially with dna storage becoming a thing, and with aspects of our physiology actually using dna modifications for their function and to "remember" things (e.g. VDJ recombination), its best not to confuse epigenetic vs genetic information storage. Each hypothesis implies different things and different followup questions.

They are certainly different things, but that's why I think more specific language should be used when the difference matters. The other comment was trying to make a more abstract point about what sort of memories an individual neuron can store using DNA, the main point wouldn't really be changed if that was through purely epigenetic mechanisms. I don't think the actual argument he made was compelling, but the claim to be discussed intended a broader definition of "in DNA" IMO. It was focused on the computational implications of the amount of storage a single neuron has.

Re: Single cortical neurons as deep artificial neural networks

#74

Post by causi »

phkahler wrote: post #71
causi wrote: post #63

Earlier quoted context omitted.

Indeed. Building an AI that matches human intelligence using equal or less mass than a human brain requires one or both of two things to be true: 1. The computational mass efficiency of brain tissue is very far from optimum. Considering the amount of time evolution has been improving upon it, I highly doubt that is true. 2. Most of the brain's computation is not involved in cognition. That may be true. We don't reall…

>> A dragonfly can take inputs from thousands of ommatidia and use them to track targets in space using only sixteen neurons. Amazing. Do you have a reference for that? I'd like to read about the findings as well as how they determined that!

Sure. https://www.pnas.org/content/110/2/696

Re: Single cortical neurons as deep artificial neural networks

#75

Post by JulianMorrison »

I'm not sure if I'm reading this right, but, they made a predictive model of a biological neuron that works? Setting aside the how, the achievement is also a thing.

How much work would it take to transcode, say, a nematode into assemblages of these things?

Re: Single cortical neurons as deep artificial neural networks

#76

Post by CodesInChaos »

magicalhippo wrote: post #69
stephencanon wrote: post #45

Earlier quoted context omitted.

> Tanh is implemented in hardware so it's used. Tanh is _not_ generally implemented in hardware, and it’s one of the fussier functions in math.h to implement well. Its only real virtues are that implementations are available everywhere, its derivative is relatively simple, and it has the right symmetries.

So sin() could be used instead of tanh, if appropriately shifted and scaled I presume?

You'd at least want to keep it at ±1 once it reaches that value instead of oscillating.

Re: Single cortical neurons as deep artificial neural networks

#77

Post by howlin »

causi wrote: post #63
zetalyrae wrote: post #2

ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…

Indeed. Building an AI that matches human intelligence using equal or less mass than a human brain requires one or both of two things to be true: 1. The computational mass efficiency of brain tissue is very far from optimum. Considering the amount of time evolution has been improving upon it, I highly doubt that is true. 2. Most of the brain's computation is not involved in cognition. That may be true. We don't reall…

Brain tissue needs to optimize for a lot of things other than computational efficiency. It needs to stay operational for decades with minimal replacement of parts and it needs to be resilient to a fair amount of bumps, diseases and chemical injury. Silicon chips don't have to be build to survive these conditions so it's possible they can be much more efficient at the computational aspect.

Re: Single cortical neurons as deep artificial neural networks

#78

Post by api »

zetalyrae wrote: post #2

ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…

Older estimates are probably short by multiple orders of magnitude.

We can see this in practice by looking at e.g. a self-driving Tesla vs. a mouse.

Watch a frightened mouse run across an uneven landscape, climbing and clambering and avoiding obstacles as it controls four independent limbs and countless small muscles in real time.

A Tesla's auto-drive is nowhere near that good in spite of having only a few levers to control: accelerate, brake, left, and right. It also has far better sensors than the mouse including better eyesight, a wider field of vision, etc., and last I checked mice do not have access to a cellular network supplying them with a heads-up macro view of the local environment. They don't have "fleet-wide learning" either.

The Tesla's AI uses over a hundred watts of power. The mouse's brain uses milliwatts.

As I wrote this my brain was consuming between 30 and 60 watts. My laptop peaks out at 80.

We are not even close to what biological neurons accomplish in raw compute, and while we are getting pretty good at training giant regression models that we call AI I am not convinced we really understand things at the algorithmic level yet either.

Re: Single cortical neurons as deep artificial neural networks

#79

Post by titzer »

laurent92 wrote: post #32
freemint wrote: post #22

> Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers I wonder how many cortical neurons it takes to approximate a ReLu or tanh well. I suspect this number being large than 1. If so the paper only shows an upper bound. Think how many neurons does it take to add a two 10 digit numbers. It is perfectly feasible that some (possibly large) part of this 5-8 layers is just "emulation overh…

That’s true, the brain uses a VM to run maths or science, only the best scientists succeed at understanding some of the concepts natively. Same for music: A student runs the music sheet in a VM, and progressively JIT makes the movements native, which allows much faster execution, and which allows building on top of the base layer. Maybe we’re doing it all wrong writing programs in assembler. We should give them to a…

> Maybe we’re doing it all wrong writing programs in assembler. We should give them to a VM,

This is what compilers do. Their input is a program in a more abstract language, either bytecode, an intermediate representation, or a source language.

The problem is that damn undecidability, which is like a minefield of rakes. It's undecidable for a compiler to tell if a program will do anything (e.g. halt). It's undecidable for a compiler to tell if two programs are equivalent. It's undecidable for a compiler to tell if a program is minimal.

So compilers have to well, be dumber. They approximate a lot.

Re: Single cortical neurons as deep artificial neural networks

#80

Post by bmh100 »

in3d wrote: post #23
danwills wrote: post #16

Earlier quoted context omitted.

Is that 256 weights per-cell per-layer, or 256 in total? Is "how many cells in each layer" a valid question too?

Their pretrained 128-wide net has a total of 9.2 million parameters: https://www.kaggle.com/selfishgene/single-neuron-as-deep-net...

9.2 million parameters per neuron. There an estimated 86 billion neurons in the human brain [0] and 19 billion in the neocortex [1]. That means that, for this strategy to emulate a human brain or neocortex would require 791 quadrillion parameters or 175 quadrillion parameters, respectively. The largest ANN built so far, GPT-3, has 175 billion parameters [2]. We are 6 orders of magnitude from being able to pull it off.

[0]: https://en.wikipedia.org/wiki/Human_brain

[1]: https://pubmed.ncbi.nlm.nih.gov/9215725/

[2]: https://siliconangle.com/2021/08/04/microsoft-researchers-re...

Post reply on HN