Single cortical neurons as deep artificial neural networks • Hacker News Board

Re: Single cortical neurons as deep artificial neural networks

#51

Post by zarzavat » Thu, Aug 12, 2021, 1:43 PM UTC

zetalyrae wrote: post #6

adrianN wrote: post #5

Earlier quoted context omitted.

Transistors, while dramatically simpler than neurons, also have fairly complicated physics. It's quite hard to determine the computational power of a microprocessor when you only know roughly how transistors work and have no idea how microprocessors work.

True, but transistors are designed by humans (for now), so you have an upper bound on complexity as well as a certain degree of modularity. Whereas biology has no such limitations because evolution doesn't care about the understandability of its designs. So I tend to err on the side of biology being more complex than not. But it's absolutely possible that the high-level behaviour is simple while the underlying implem…

Complexity cuts both ways. Complexity, as every programmer knows, is no friend, and can easily create more problems than it solves.

Often the most efficient solution to a problem ends up being a simple one. It would be very surprising if the human brain's Rube Goldberg machine was anywhere near close to a mathematically optimal implementation of intelligence.

It's possible that much of the complexity in a biological neuron is simply working around other complexity introduced by biology, solving problems that we programmers do not even have to think about because we can simply directly use matrix multiplication.

Re: Single cortical neurons as deep artificial neural networks

#52

Post by hyper_dynamics » Thu, Aug 12, 2021, 1:44 PM UTC

adrianN wrote: post #40

vlovich123 wrote: post #36

Earlier quoted context omitted.

How so?

Suppose you write a program that simulates a TM and then prints "Halt" when the TM halts. The magic AI could optimize this program to just a print (or an infinite loop). This requires solving the halting problem.

It might not be able to do it for any turing machine/ a universal turing machine – but it might quickly figure out what a turing machine will do without executing all steps of it.

Re: Single cortical neurons as deep artificial neural networks

#53

Post by simiones » Thu, Aug 12, 2021, 1:46 PM UTC

freemint wrote: post #22

> Cortical neurons are well approximated by a deep neural network (DNN) with 5–8 layers I wonder how many cortical neurons it takes to approximate a ReLu or tanh well. I suspect this number being large than 1. If so the paper only shows an upper bound. Think how many neurons does it take to add a two 10 digit numbers. It is perfectly feasible that some (possibly large) part of this 5-8 layers is just "emulation overh…

On the other hand, a single cell much simpler than a neuron (any bacteria) is able to perform significantly more complex calculations than any ANN we've tried so far (successfully interacting with an environment to move and find food).

Comparing these kinds of disparate tasks for "computational power levels" between vastly different architectures one of which we're not even close to understanding is generally pretty futile.

Re: Single cortical neurons as deep artificial neural networks

#54

Post by shmageggy » Thu, Aug 12, 2021, 1:48 PM UTC

l33t2328 wrote: post #47

nabla9 wrote: post #15

Earlier quoted context omitted.

Your terminology is inaccurate. Threshold neurons like Pitts-McCulloch neurons are rarely used in artificial neural network architectures. They can't learn. ANN's uses a nonlinear activation function that is differentiable (at least in practice). That small change makes huge difference.

> They can't learn Is there a good (i.e. theoretical) reason for this?

No, because it's wrong. Threshold neurons are still differentiable almost everywhere, no different than Relus which are ubiquitous. They may not be very good activation functions but they don't prevent a network from learning.

Re: Single cortical neurons as deep artificial neural networks

#55

Post by simiones » Thu, Aug 12, 2021, 1:49 PM UTC

hyper_dynamics wrote: post #52

adrianN wrote: post #40

Earlier quoted context omitted.

Suppose you write a program that simulates a TM and then prints "Halt" when the TM halts. The magic AI could optimize this program to just a print (or an infinite loop). This requires solving the halting problem.

It might not be able to do it for any turing machine/ a universal turing machine – but it might quickly figure out what a turing machine will do without executing all steps of it.

It might get it right some of the time, but it will be necessarily wrong some of the time. It's also very possible that for many (possibly even most) TMs, the most efficient algorithm for predicting the output is that TM itself.

Re: Single cortical neurons as deep artificial neural networks

#56

Post by caddemon » Thu, Aug 12, 2021, 1:50 PM UTC

twic wrote: post #38

nootropicat wrote: post #26

Earlier quoted context omitted.

To add to that, there's evidence that may mean that memories are stored in DNA, shuffled between neurons in RNA capsids, evolutionary borrowed from retroviruses. That would make each neuron a complicated computer, functionally equivalent to one pc in capability and data storage. "The neuronal gene Arc is essential for long-lasting information storage in the mammalian brain, mediates various forms of synaptic plastici…

The passage you quote very much does not amount to "memories are stored in DNA", and as far as i know, memories are not stored in DNA sequence, and would be very surprised if they were. What this mechanism does is take the transcription level of a gene in one cell, and induce a proportional signal in another cell. There are numerous mechanisms which do that in cells. Most of them are nowhere near as weird as Arc, but…

It's arguable that some memories are stored via epigenetic modifications to DNA, depending on how you define "storing memories". It is very unlikely it is storing memories in the way humans typically think of memories. But I don't think "in DNA" necessarily implies "in DNA sequence".

Re: Single cortical neurons as deep artificial neural networks

#57

Post by simiones » Thu, Aug 12, 2021, 1:55 PM UTC

raducu wrote: post #20

hoseja wrote: post #9

Earlier quoted context omitted.

That doesn't follow at all. Human neurons aren't much different from mouse neurons, maybe chicken neurons, mosquito neurons. After you faithfully model a neuron you still need untold myriads of them and their interconnects to get human brain.

Also, sensations and feelings are not a logical/mathematical byproduct of the neurons; no matter how "well" you simulate "neurons", feelings and sensations will not emerge.

Unless you believe in a transcendent soul that could be the source of these sensations or feelings, this assertion doesn't make sense. Assuming there is no supernatural soul, it's logically impossible for anything humans experience to not arise from the human body.

This entire notion of qualia is a philosophical quagmire predicated on the idea that if you can imagine something, it must be true ("we can imagine a zombie that behaves exactly like a human, but doesn't have qualia at all"). It's actually as laughable as the "argument from perfection" for the existence of a god.

Re: Single cortical neurons as deep artificial neural networks

#58

Post by ben_w » Thu, Aug 12, 2021, 1:57 PM UTC

qayxc wrote: post #43

mrfusion wrote: post #37

Earlier quoted context omitted.

Don’t forget that transistors are millions of times faster too.

Could indeed be a trade-off between speed vs complexity and efficiency. Maybe there will be transistor-based human-level AGI soon, but I guess it would require several kilowatts of power compared to the 20 or so watts a human brain requires.

The lowest lowball estimate I’ve seen for the computational capacity of the human brain is 20e15 FLOPS (Kurzweil, 1999), and that would require 2/3 of a megawatt with the current best ranked supercomputer on the Green500 list.

There are already several computers more powerful than this that don’t appear close to a working full brain emulation, and we definitely don’t understand intelligence well enough yet to engineer something like ourselves, so it’s reasonable to be skeptical of estimates saying we’ll be at the kilowatts level “soon” even if it turns out we’re just missing a step which will be obvious in hindsight.

(Unless by “soon” you mean 15 years; I don’t want to bet on anything on that timescale).

Re: Single cortical neurons as deep artificial neural networks

#59

Post by titzer » Thu, Aug 12, 2021, 1:58 PM UTC

yarcob wrote: post #13

adrianN wrote: post #5

Earlier quoted context omitted.

Transistors, while dramatically simpler than neurons, also have fairly complicated physics. It's quite hard to determine the computational power of a microprocessor when you only know roughly how transistors work and have no idea how microprocessors work.

But digital chips are designed to use transistors as on/off switches, so the complicated physics don't matter for understanding a microprocessor. (It's different for analog ICs, where the characteristics of the individual transistors do matter)

Wouldn't it suck if brains are just reservoirs used by the internet that can store roughly 3 memes at a time?

Re: Single cortical neurons as deep artificial neural networks

#60

Post by lolthishuman » Thu, Aug 12, 2021, 2:06 PM UTC

caddemon wrote: post #56

twic wrote: post #38

Earlier quoted context omitted.

The passage you quote very much does not amount to "memories are stored in DNA", and as far as i know, memories are not stored in DNA sequence, and would be very surprised if they were. What this mechanism does is take the transcription level of a gene in one cell, and induce a proportional signal in another cell. There are numerous mechanisms which do that in cells. Most of them are nowhere near as weird as Arc, but…

It's arguable that some memories are stored via epigenetic modifications to DNA, depending on how you define "storing memories". It is very unlikely it is storing memories in the way humans typically think of memories. But I don't think "in DNA" necessarily implies "in DNA sequence".

For whatever it’s worth, similar conclusions are shared in ayahuasca rituals. The source of many journeys are reconnecting with the memory within DNA. I’ve had similar intuitions and heard this from more than one source that has partaked.