Single cortical neurons as deep artificial neural networks • Hacker News Board

Re: Single cortical neurons as deep artificial neural networks

#61

Post by sdenton4 » Thu, Aug 12, 2021, 2:08 PM UTC

nabla9 wrote: post #34

freemint wrote: post #22

> Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers I wonder how many cortical neurons it takes to approximate a ReLu or tanh well. I suspect this number being large than 1. If so the paper only shows an upper bound. Think how many neurons does it take to add a two 10 digit numbers. It is perfectly feasible that some (possibly large) part of this 5-8 layers is just "emulation overh…

There is no need to approximate a ReLu or tanh well. Machine learning is statistical. The accuracy of these functions is not that important ReLu is buggy and has an incorrect activation function for deep learning because it's not continuous everywhere. In practice, it rarely matters. It's chosen only because it's fast to implement buggy function than use someting proper. The exact shape of tanh is not important eithe…

Meanwhile, if we actually understood brains, I bet we would find endless examples of 'improper' behavior. Evolution picks up what seems to work, and sloooowly improves the parts that break, leaving good enough alone. (After all, if it doesn't affect reproductive probabilities, it doesn't matter.)

Activation functions will almost certainly not be the crux move for solving AGI.

Re: Single cortical neurons as deep artificial neural networks

#62

Post by jerf » Thu, Aug 12, 2021, 2:16 PM UTC

zetalyrae wrote: post #6

adrianN wrote: post #5

Earlier quoted context omitted.

Transistors, while dramatically simpler than neurons, also have fairly complicated physics. It's quite hard to determine the computational power of a microprocessor when you only know roughly how transistors work and have no idea how microprocessors work.

True, but transistors are designed by humans (for now), so you have an upper bound on complexity as well as a certain degree of modularity. Whereas biology has no such limitations because evolution doesn't care about the understandability of its designs. So I tend to err on the side of biology being more complex than not. But it's absolutely possible that the high-level behaviour is simple while the underlying implem…

Probably more to the point is that humans are using transistors in their designs, so we deliberately confine them to their simple modes. It is an oversimplification to talk about transistors being either "on" or "off" because they technically have all sorts of intermediate states, but with a few exceptions, human designs avoid those intermediate states like the plague, because A: they defy our ability to build logic with them and B: depending on those exact behaviors means we can't mass-manufacture chips because the variance of the exact behaviors will be too high.

In principle one could imagine a processor design that works on these intermediate states that somehow vastly exceeds the computational power of a modern system despite using the same base transistors; in practice we have no idea how to build such a thing, and if we did, we wouldn't know how to build a second one of the same thing reliably either.

Biology lacks this restriction. That doesn't mean it's pure and utter chaos, either, there's bounds on that because it still needs systems to at least be metastable. But where humans engineer almost exclusively with stable systems, biology freely uses metastable systems all over the place. And then, even more remarkably, it deals with the question of how to replicate such a strange system in a way that no modern human engineer ever would by making every instance unique, and still somehow functional.

It's a tough act to follow.

Re: Single cortical neurons as deep artificial neural networks

#63

Post by causi » Thu, Aug 12, 2021, 2:21 PM UTC

zetalyrae wrote: post #2

ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…

Indeed. Building an AI that matches human intelligence using equal or less mass than a human brain requires one or both of two things to be true: 1. The computational mass efficiency of brain tissue is very far from optimum. Considering the amount of time evolution has been improving upon it, I highly doubt that is true. 2. Most of the brain's computation is not involved in cognition. That may be true. We don't really know.

There are hard limits. No matter how you try you can't perfectly simulate three atoms using two atoms. If it turns out we have to, in software, represent fifty percent of neuronal activity to create consciousness we're in real trouble. A dragonfly can take inputs from thousands of ommatidia and use them to track targets in space using only sixteen neurons. How many transistors would it take us to do the same? Take that ratio and apply it to the 86 billion neurons in the human brain and you have a rough idea of what it will take to create strong AI. The numbers aren't promising.

Re: Single cortical neurons as deep artificial neural networks

#64

Post by vikramkr » Thu, Aug 12, 2021, 2:21 PM UTC

caddemon wrote: post #56

twic wrote: post #38

Earlier quoted context omitted.

The passage you quote very much does not amount to "memories are stored in DNA", and as far as i know, memories are not stored in DNA sequence, and would be very surprised if they were. What this mechanism does is take the transcription level of a gene in one cell, and induce a proportional signal in another cell. There are numerous mechanisms which do that in cells. Most of them are nowhere near as weird as Arc, but…

It's arguable that some memories are stored via epigenetic modifications to DNA, depending on how you define "storing memories". It is very unlikely it is storing memories in the way humans typically think of memories. But I don't think "in DNA" necessarily implies "in DNA sequence".

In dna definitely implies in dna sequence. Especially with dna storage becoming a thing, and with aspects of our physiology actually using dna modifications for their function and to "remember" things (e.g. VDJ recombination), its best not to confuse epigenetic vs genetic information storage. Each hypothesis implies different things and different followup questions.

Re: Single cortical neurons as deep artificial neural networks

#65

Post by sdenton4 » Thu, Aug 12, 2021, 2:24 PM UTC

causi wrote: post #63

zetalyrae wrote: post #2

ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…

Indeed. Building an AI that matches human intelligence using equal or less mass than a human brain requires one or both of two things to be true: 1. The computational mass efficiency of brain tissue is very far from optimum. Considering the amount of time evolution has been improving upon it, I highly doubt that is true. 2. Most of the brain's computation is not involved in cognition. That may be true. We don't reall…

/2. Most of the brain's computation is not involved in cognition. That may be true. We don't really know./

I thought this at least was fairly well understood: We do in fact use our whole brains, as anything less would be a fantastic waste of resources, which evolution would have taken care of long ago. We have numerous human-specific adaptations to deal with the relatively massive brains we're carrying around.

Re: Single cortical neurons as deep artificial neural networks

#66

Post by sdenton4 » Thu, Aug 12, 2021, 2:29 PM UTC

shmageggy wrote: post #54

l33t2328 wrote: post #47

Earlier quoted context omitted.

> They can't learn Is there a good (i.e. theoretical) reason for this?

No, because it's wrong. Threshold neurons are still differentiable almost everywhere, no different than Relus which are ubiquitous. They may not be very good activation functions but they don't prevent a network from learning.

Sorry, but I think I disagree...

If I understand the PM neuron, it's outputs are boolean. This means there's no backprop signal, despite differentiability, since the outputs are constant (thus, gradient zero) in any neighborhood, which in turns zeros out any learning signal you would want to backprop through them. So you need a different learning strategy than backprop to use them.

(see also: the 'dead neuron' problem/phenomenon with ReLU activations.)

Re: Single cortical neurons as deep artificial neural networks

#67

Post by causi » Thu, Aug 12, 2021, 2:30 PM UTC

sdenton4 wrote: post #65

causi wrote: post #63

Earlier quoted context omitted.

Indeed. Building an AI that matches human intelligence using equal or less mass than a human brain requires one or both of two things to be true: 1. The computational mass efficiency of brain tissue is very far from optimum. Considering the amount of time evolution has been improving upon it, I highly doubt that is true. 2. Most of the brain's computation is not involved in cognition. That may be true. We don't reall…

/2. Most of the brain's computation is not involved in cognition. That may be true. We don't really know./ I thought this at least was fairly well understood: We do in fact use our whole brains, as anything less would be a fantastic waste of resources, which evolution would have taken care of long ago. We have numerous human-specific adaptations to deal with the relatively massive brains we're carrying around.

When you're building an AI you may not need the neurons involved with, for example, breathing. That's what I'm talking about. I'm not a neuroscientist so I don't know for sure whether all the neurons we use for muscle and organ control do double-duty to help us cogitate.

Re: Single cortical neurons as deep artificial neural networks

#68

Post by fnord77 » Thu, Aug 12, 2021, 2:31 PM UTC

zetalyrae wrote: post #2

ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…

what's also interesting is non-neuronal computational abilities. Allosteric regulation on receptors chained together seem like they have computational abilities. And they're on all sorts of cells

I'm sure there's others.

Re: Single cortical neurons as deep artificial neural networks

#69

Post by magicalhippo » Thu, Aug 12, 2021, 2:32 PM UTC

stephencanon wrote: post #45

nabla9 wrote: post #34

Earlier quoted context omitted.

There is no need to approximate a ReLu or tanh well. Machine learning is statistical. The accuracy of these functions is not that important ReLu is buggy and has an incorrect activation function for deep learning because it's not continuous everywhere. In practice, it rarely matters. It's chosen only because it's fast to implement buggy function than use someting proper. The exact shape of tanh is not important eithe…

> Tanh is implemented in hardware so it's used. Tanh is _not_ generally implemented in hardware, and it’s one of the fussier functions in math.h to implement well. Its only real virtues are that implementations are available everywhere, its derivative is relatively simple, and it has the right symmetries.

So sin() could be used instead of tanh, if appropriately shifted and scaled I presume?

Re: Single cortical neurons as deep artificial neural networks

#70

Post by gnramires » Thu, Aug 12, 2021, 2:32 PM UTC

zetalyrae wrote: post #2

ANN neurons are Pitts-McCulloch neurons, an extremely stylized model of the 1940's understanding of neurons. Each neuron represents a dot product plus a function application. A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms. A single synapse is a huge landscape studded with receptors of various kinds, and the whole thing is swimming in salt solution where chemicals diffuse stochastical…

> A biological neuron is 6e17 Daltons[0], so on the order of quadrillions of atoms

You may be implying there could be information stored in all those atoms, but I'm not sure that's possible. We live in a thermal bath, this means the behavior of atoms is usually stochastic. Their position cannot reliably hold information without dispersing it rapidly into the thermal environment. One way to get around this is to form chemical bonds, like in the DNA, where some kind of order or position is stable. But it necessitates this chemical structure, and importantly it necessitates as well a chemical reading mechanism and a chemical writing mechanism (or some kind of kinetic activation). Most parts of a cell are not prepared for any of that as far as I can tell. Information really should be carried by discrete elements such as neurotransmitters, as well as continuous but temporary (unstable) elements such as electric impulses (potentially caused by complex responses to electric potential and current inside the neuron). Even in the electric case, the fact about information stands; electric state is also encoded in atoms.

In other words, we almost certainly don't need the full fidelity to reproduce the behavior of a neuron.

I would need more rigorous examination of the neuron to give a confident estimate, but as a rule of thumb the concentration of relevant information everywhere is much less than DNA's (and mostly negligible everywhere) -- certainly a very interesting research program.

From a quick googling, E. Coli DNA has about 4.6 x 10^6 bp, so I would be reasonably confident in an upper bound to neuron information as (volume of neuron/volume of e. coli) x 5 x 10^6 bits (i.e. 1 megabit, 125 kb). The reality is probably much less. DNA is so dense because the reading and replication time is relatively slow. It's not made for rapid, random access at the speed of thought.

If I were to guess, I'd say long term information is probably retained within the concentration of compounds that can be read electro-chemically. The question of information then is how sensitive the neural system as a whole is to differences in concentration and differences in timing and amplitude of neural impulses. Again given thermal noise in the brain and limitations of amplitude, you can give strict upper bounds on neural communication (I'd be surprised at sensitivities more than a few ppm).

So essentially

1 neuron And also

1 spike Again, I'd require more information on spikes, but they carry maybe 20-40 bits at most -- so not more than a double f.p., although in ANNs again because of low sensitivities due to architecture most of LSB information doesn't contribute significantly to the computation (whereas the brain could multiplex information more effectively). So it's still likely in the order of 1 spike <= 10 flops.