Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Single cortical neurons as deep artificial neural networks

sciencedirect.com

Post reply on HN
Print view View original

101–110 of 135 posts

Re: Single cortical neurons as deep artificial neural networks

#101

Post by stephencanon »

CodesInChaos wrote: post #86
stephencanon wrote: post #45

Earlier quoted context omitted.

> Tanh is implemented in hardware so it's used. Tanh is _not_ generally implemented in hardware, and it’s one of the fussier functions in math.h to implement well. Its only real virtues are that implementations are available everywhere, its derivative is relatively simple, and it has the right symmetries.

You're right that neural networks don't care too much the shape of most activation functions. I assume that splicing together two decaying exponential functions at the origin would work just as well in practice. However tanh is a bit more special than just having the right symmetries. Sigmoid is the correct function to turn an additive value into a probability (range 0 to 1). Tanh is a scaled sigmoid which fulfills t…

Yeah, wiki has a decent survey of sigmoid (the family, not the specific function ML people often refer to by that name) functions here: https://en.wikipedia.org/wiki/Sigmoid_function#/media/File:G...

Note that tanh saturates to ±1 faster than most except erf when normalized to have slope 1 at the origin (its series at +infinity is like 1 - 2e^{-2x} + o(e^{-4x}), while many of the other options have polynomial series, so they don't approach 1 nearly as fast).

I suspect some applications would in theory rather use erf, but erf is even worse to compute than tanh (on the other hand, erf's derivative is really nice, so who knows?)

Re: Single cortical neurons as deep artificial neural networks

#102

Post by caddemon »

vikramkr wrote: post #99
caddemon wrote: post #73

Earlier quoted context omitted.

They are certainly different things, but that's why I think more specific language should be used when the difference matters. The other comment was trying to make a more abstract point about what sort of memories an individual neuron can store using DNA, the main point wouldn't really be changed if that was through purely epigenetic mechanisms. I don't think the actual argument he made was compelling, but the claim…

Perhaps, but I'd argue the differences do matter quite a bit to the computational implications, especially as it relates to ability to reprogram, state space, and persistence, and I think there would be a much different reaction if I made a post mixing up data on a computer's hard drive with the state of whether each transistor in a cpu is on or off, or something like that.

There are different types of epigenetic mechanisms anyway which have different levels of persistence/reprogrammability, so to dive into technical details one would need to get very specific. Different methods for changing the DNA sequence would also have functional differences in how they could store a memory. And it's important to distinguish these mechanisms on a single cell level versus how they function in the body as a whole.

In a single cell I wouldn't call VDJ recombination memory, it's just an efficient way of encoding many different possibilities for types of receptors (and then selecting one). There exist many cells in your body that match a potential antigen you've never seen before - it's just that there are an extremely small number of them. The memory is really encoded by increasing the population of that cell in the body. Which has different storage properties than the individual cell's DNA does. Epigenetic change to a single neuron is unlikely to have a functional effect either, but it is a change that can occur to a mature cell as a reaction to the environment, in a way that VDJ recombination is not. AFAIK there is a lot left to be understood about how epigenetic modifications affect the brain.

I agree this is all interesting and can have implications for computational models, but there are models at many levels of abstraction. So I don't think it is necessary to get at the biological details in order to discuss higher level computational implications. Questions about the capabilities of the memory for each individual neuron would arise, but they could be theorized about without much knowledge of biological details. Of course it depends what your goals are whether you would consider that useful.

Anyway, I think we're both in agreement the OG comment was wrong, I just feel that "in DNA" is very ambiguous, and also not really the problem with his comment given the purpose of HN.

Re: Single cortical neurons as deep artificial neural networks

#103

Post by cblconfederate »

JulianMorrison wrote: post #75

I'm not sure if I'm reading this right, but, they made a predictive model of a biological neuron that works? Setting aside the how, the achievement is also a thing. How much work would it take to transcode, say, a nematode into assemblages of these things?

Compartmental modeling is the standard method for simulating neural tissue by breaking it down to electrical cylinders. There is already openworm.org which is a full cell-by-cell simulation of the c.elegans

Larger projects like the Human Brain Project exist, but despite the existence of large scale compartmental simulations, we ve gained little insight about how brains work

Re: Single cortical neurons as deep artificial neural networks

#104

Post by qayxc »

ben_w wrote: post #58
qayxc wrote: post #43

Earlier quoted context omitted.

Could indeed be a trade-off between speed vs complexity and efficiency. Maybe there will be transistor-based human-level AGI soon, but I guess it would require several kilowatts of power compared to the 20 or so watts a human brain requires.

The lowest lowball estimate I’ve seen for the computational capacity of the human brain is 20e15 FLOPS (Kurzweil, 1999), and that would require 2/3 of a megawatt with the current best ranked supercomputer on the Green500 list. There are already several computers more powerful than this that don’t appear close to a working full brain emulation, and we definitely don’t understand intelligence well enough yet to enginee…

> Unless by “soon” you mean 15 years; I don’t want to bet on anything on that timescale

Well, I'm old so by "soon" I mean within my lifetime, e.g. the next 3 to 4 decades.

edit: to clarify a little, when the term AI was first coined by McCarthy in 1956, researchers were confident in cracking AGI within a decade. Then AI-Winter came and people became more cautious. So when I say "soon", I mean it's probably not going to be another 65 years, but also not 5 months or 5 years (unexpected breakthroughs aside).

Re: Single cortical neurons as deep artificial neural networks

#105

Post by runT1ME »

simiones wrote: post #55
hyper_dynamics wrote: post #52

Earlier quoted context omitted.

It might not be able to do it for any turing machine/ a universal turing machine – but it might quickly figure out what a turing machine will do without executing all steps of it.

It might get it right some of the time, but it will be necessarily wrong some of the time. It's also very possible that for many (possibly even most) TMs, the most efficient algorithm for predicting the output is that TM itself.

It might only be wrong for programs that have never existed and will ever exist.

Re: Single cortical neurons as deep artificial neural networks

#106

Post by MauranKilom »

titzer wrote: post #79
laurent92 wrote: post #32

Earlier quoted context omitted.

That’s true, the brain uses a VM to run maths or science, only the best scientists succeed at understanding some of the concepts natively. Same for music: A student runs the music sheet in a VM, and progressively JIT makes the movements native, which allows much faster execution, and which allows building on top of the base layer. Maybe we’re doing it all wrong writing programs in assembler. We should give them to a…

> Maybe we’re doing it all wrong writing programs in assembler. We should give them to a VM, This is what compilers do. Their input is a program in a more abstract language, either bytecode, an intermediate representation, or a source language. The problem is that damn undecidability, which is like a minefield of rakes. It's undecidable for a compiler to tell if a program will do anything (e.g. halt). It's undecidabl…

> The problem is that damn undecidability, which is like a minefield of rakes. It's undecidable for a compiler to tell if a program will do anything (e.g. halt). It's undecidable for a compiler to tell if two programs are equivalent. It's undecidable for a compiler to tell if a program is minimal.

Only for Turing complete languages, to be clear. Now, of course, most interesting problems cannot really be solved in sub-Turing languages, but it's still a fundamental point to consider.

Re: Single cortical neurons as deep artificial neural networks

#107

Post by MauranKilom »

CodesInChaos wrote: post #86
stephencanon wrote: post #45

Earlier quoted context omitted.

> Tanh is implemented in hardware so it's used. Tanh is _not_ generally implemented in hardware, and it’s one of the fussier functions in math.h to implement well. Its only real virtues are that implementations are available everywhere, its derivative is relatively simple, and it has the right symmetries.

You're right that neural networks don't care too much the shape of most activation functions. I assume that splicing together two decaying exponential functions at the origin would work just as well in practice. However tanh is a bit more special than just having the right symmetries. Sigmoid is the correct function to turn an additive value into a probability (range 0 to 1). Tanh is a scaled sigmoid which fulfills t…

I assume that splicing together two decaying exponential functions at the origin would work just as well in practice.

Also known as tanh: https://en.wikipedia.org/wiki/Hyperbolic_functions

One "disadvantage" is that it doesn't saturate to [-1.0, 1.0] like appropriately scaled tanh.

Re: Single cortical neurons as deep artificial neural networks

#108

Post by MauranKilom »

simiones wrote: post #55
hyper_dynamics wrote: post #52

Earlier quoted context omitted.

It might not be able to do it for any turing machine/ a universal turing machine – but it might quickly figure out what a turing machine will do without executing all steps of it.

It might get it right some of the time, but it will be necessarily wrong some of the time. It's also very possible that for many (possibly even most) TMs, the most efficient algorithm for predicting the output is that TM itself.

> It might get it right some of the time, but it will be necessarily wrong some of the time

Yes, but the proof of the halting problem relies on diagonalization - i.e. a very exotic and carefully crafted input.

I would also like to note that, analogously, modern SAT solvers can solve most instances of NP-hard problems in polynomial time. Even though there exist hard cases they cannot solve in polynomial time (well, assuming P != NP), in practice these polynomial algorithms are exceedingly useful.

> It's also very possible that for many (possibly even most) TMs, the most efficient algorithm for predicting the output is that TM itself.

That could very well be true, but might still not hold for the subset of inputs (i.e. programs) that we actually care about in practice.

Relatedly, Kolmogorov complexity makes for some very interesting further reading (which essentially formalizes the problem we're discussing here): https://en.wikipedia.org/wiki/Kolmogorov_complexity

Re: Single cortical neurons as deep artificial neural networks

#109

Post by dwd »

tambourine_man wrote: post #93

The recent episode of the Lex Fridman podcast with Jeff Hawkins touches on this issue of the computational capabilities of single neurons: https://youtube.com/watch?v=Z1KwkpTUbkg Great episode all around.

He also appeared recently on Sam Harris's podcast to a bit more more skepticism.

I thought he didn't break much new ground from his previous appearance on Lex's podcast in 2019, which I went back and listened to before listening to the latest one.

But...in saying that - listening to him again triggered a multitude of aha moments for me, and got me thinking about things like: "what is a dream", "how does deja-vu work" in the context of cortical columns, 1000 brain theory, predictive voting, etc.

I though his most recent explanation on how creating an artificial neocortex is not a risk in itself, but rather what you ask it to do. He was also adamant that giving an AI the ability to self-replicate should be where we draw the line.

This was very interesting when they previously discussed that topic but was months before SARS-CoV-2:

https://www.happyscribe.com/public/lex-fridman-podcast-artif...

Re: Single cortical neurons as deep artificial neural networks

#110

Post by mr_toad »

JulianMorrison wrote: post #75

I'm not sure if I'm reading this right, but, they made a predictive model of a biological neuron that works? Setting aside the how, the achievement is also a thing. How much work would it take to transcode, say, a nematode into assemblages of these things?

> How much work would it take to transcode, say, a nematode into assemblages of these things?

You also need to find a way to simulate input (an environment) for this simulated “brain”, otherwise I suspect it would suffer the problems associated with sensory deprivation.

Post reply on HN