Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single cortical neurons as deep artificial neural networks

sciencedirect.com

Post reply on HN
Print view View original

41–50 of 135 posts

Re: Single cortical neurons as deep artificial neural networks

#41

Post by neuah »

freemint wrote: post #22

> Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers I wonder how many cortical neurons it takes to approximate a ReLu or tanh well. I suspect this number being large than 1. If so the paper only shows an upper bound. Think how many neurons does it take to add a two 10 digit numbers. It is perfectly feasible that some (possibly large) part of this 5-8 layers is just "emulation overh…

I guess it depends on how accurately you're thinking about those functions being approximated. Neurons have a natural nonlinearity to their input-output (transfer) function, most obvious of which is the action potential threshold. Biological neurons have a saturating nonlinearity because there is an upper limit on their firing rate, but in certain regimes the nonlinearity of a single neuron could easily look qualitatively similar to relu or a (non-negative) tanh.

Re: Single cortical neurons as deep artificial neural networks

#42

Post by greens »

zetalyrae wrote: post #11
singularity2001 wrote: post #8

Earlier quoted context omitted.

Not to speak of all the 'weights' which are stored internaly in DNA methylation, among other memory forming mechanisms (proven recently). (part of the quadrillion atoms system of course)

And if you need ab-initio quantum chemistry (or equivalent) simulations of every atom in a human brain, the timelines for both AGI and whole brain emulation get pushed far, far into the future. Assuming AGI requires about as much hardware as it'd take to emulate a brain, at least.

My 2c (apologies for the aggressive tone -- I'm just excited about AGI):

That's a very very weak upper bound on how much hardware it takes. I think it's not all that different from emulating a Nintendo64 with a quantum simulation of the hardware.

For complex systems to work (not to mention evolve), they need to be robust to small perturbations -- there's no way the computation the brain is doing is sensitive to the details of particular atoms. There has to be redundancy, modularity, etc. These things aren't human inventions so much as they are the only way to meaningfully move in a 2^|giant-number| state-space.

Re: Single cortical neurons as deep artificial neural networks

#43

Post by qayxc »

mrfusion wrote: post #37
adrianN wrote: post #5

Earlier quoted context omitted.

Transistors, while dramatically simpler than neurons, also have fairly complicated physics. It's quite hard to determine the computational power of a microprocessor when you only know roughly how transistors work and have no idea how microprocessors work.

Don’t forget that transistors are millions of times faster too.

Could indeed be a trade-off between speed vs complexity and efficiency.

Maybe there will be transistor-based human-level AGI soon, but I guess it would require several kilowatts of power compared to the 20 or so watts a human brain requires.

Re: Single cortical neurons as deep artificial neural networks

#44

Post by zetalyrae »

greens wrote: post #42
zetalyrae wrote: post #11

Earlier quoted context omitted.

And if you need ab-initio quantum chemistry (or equivalent) simulations of every atom in a human brain, the timelines for both AGI and whole brain emulation get pushed far, far into the future. Assuming AGI requires about as much hardware as it'd take to emulate a brain, at least.

My 2c (apologies for the aggressive tone -- I'm just excited about AGI): That's a very very weak upper bound on how much hardware it takes. I think it's not all that different from emulating a Nintendo64 with a quantum simulation of the hardware. For complex systems to work (not to mention evolve), they need to be robust to small perturbations -- there's no way the computation the brain is doing is sensitive to the d…

DNA/RNA is near-atomic level storage. There's redundancy in that there are multiple instances of each neuron|receptor|etc.

Re: Single cortical neurons as deep artificial neural networks

#45

Post by stephencanon »

nabla9 wrote: post #34
freemint wrote: post #22

> Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers I wonder how many cortical neurons it takes to approximate a ReLu or tanh well. I suspect this number being large than 1. If so the paper only shows an upper bound. Think how many neurons does it take to add a two 10 digit numbers. It is perfectly feasible that some (possibly large) part of this 5-8 layers is just "emulation overh…

There is no need to approximate a ReLu or tanh well. Machine learning is statistical. The accuracy of these functions is not that important ReLu is buggy and has an incorrect activation function for deep learning because it's not continuous everywhere. In practice, it rarely matters. It's chosen only because it's fast to implement buggy function than use someting proper. The exact shape of tanh is not important eithe…

> Tanh is implemented in hardware so it's used.

Tanh is _not_ generally implemented in hardware, and it’s one of the fussier functions in math.h to implement well. Its only real virtues are that implementations are available everywhere, its derivative is relatively simple, and it has the right symmetries.

Re: Single cortical neurons as deep artificial neural networks

#46

Post by kortex »

hoseja wrote: post #9
londons_explore wrote: post #3

This is trying to train a neural network to match the behaviour of various mathematical models of biological neurons. Said models still don't seem to be very close to being able to predict the behaviour of biological neurons though. If they did, then we'd have invented artificial human-level intelligence.

That doesn't follow at all. Human neurons aren't much different from mouse neurons, maybe chicken neurons, mosquito neurons. After you faithfully model a neuron you still need untold myriads of them and their interconnects to get human brain.

> Human neurons aren't much different from mouse neurons

We don't actually know this. Yeah the cerebellum and substantia nigra and other regions preserved across mammals are probably conserved in the neural structure as well. But the human neocortex has quite radically different gene expression compared to rats (which results in the morphological differences). There very well could be "more processing power per neuron" in humans vs rodents.

Re: Single cortical neurons as deep artificial neural networks

#47

Post by l33t2328 »

nabla9 wrote: post #15
zetalyrae wrote: post #2

ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…

Your terminology is inaccurate. Threshold neurons like Pitts-McCulloch neurons are rarely used in artificial neural network architectures. They can't learn. ANN's uses a nonlinear activation function that is differentiable (at least in practice). That small change makes huge difference.

> They can't learn

Is there a good (i.e. theoretical) reason for this?

Re: Single cortical neurons as deep artificial neural networks

#48

Post by short_sells_poo »

qayxc wrote: post #43
mrfusion wrote: post #37

Earlier quoted context omitted.

Don’t forget that transistors are millions of times faster too.

Could indeed be a trade-off between speed vs complexity and efficiency. Maybe there will be transistor-based human-level AGI soon, but I guess it would require several kilowatts of power compared to the 20 or so watts a human brain requires.

> Maybe there will be transistor-based human-level AGI soon, but I guess it would require several kilowatts of power compared to the 20 or so watts a human brain requires

Given that entire datacenters don't come close to an AGI, let alone a human level one, I fear "several kilowatts of power" is lowballing it by a significant number of orders of magnitude.

More to the point, we don't even know how or what gives rise to a general intelligence, and even defining it is basically a philosophical question. To me, the optimism of some AI enthusiasts (and I don't mean specifically the parent) feels like cavemen contemplating an expedition to the stars shortly after they invented the sling.

I think the chances of us arriving to anything close to an AGI iteratively based on our current capabilities is a pipe dream.

Re: Single cortical neurons as deep artificial neural networks

#49

Post by Arech »

l33t2328 wrote: post #47
nabla9 wrote: post #15

Earlier quoted context omitted.

Your terminology is inaccurate. Threshold neurons like Pitts-McCulloch neurons are rarely used in artificial neural network architectures. They can't learn. ANN's uses a nonlinear activation function that is differentiable (at least in practice). That small change makes huge difference.

> They can't learn Is there a good (i.e. theoretical) reason for this?

Non-differentiability, obviously

Re: Single cortical neurons as deep artificial neural networks

#50

Post by ben_w »

zetalyrae wrote: post #6
adrianN wrote: post #5

Earlier quoted context omitted.

Transistors, while dramatically simpler than neurons, also have fairly complicated physics. It's quite hard to determine the computational power of a microprocessor when you only know roughly how transistors work and have no idea how microprocessors work.

True, but transistors are designed by humans (for now), so you have an upper bound on complexity as well as a certain degree of modularity. Whereas biology has no such limitations because evolution doesn't care about the understandability of its designs. So I tend to err on the side of biology being more complex than not. But it's absolutely possible that the high-level behaviour is simple while the underlying implem…

Transistors were invented by humans, but chip layouts themselves involve a lot of automation and at more than one level — Verilog and VHDL both appeared in the 1980s, and there’s tools at both higher and lower levels of abstraction than those two.
Post reply on HN