Live data from Hacker News

How deep is the brain? The shallow brain hypothesis

nature.com

141–150 of 183 posts

Re: How deep is the brain? The shallow brain hypothesis

#141

Earlier quoted context omitted.

> They can theoretically model any function, but the number of parameters needed means in practice they can't. Even theoretically, no they can't. They can theoretically model any continuos function. Plus, even for continuous functions, the theorem only proves that, for any function, there exists some NN that approximates it to arbitrary precision. It is not known whether there is some base NN + finite training set th…

I'm not sure it is all that interesting of a distinction seeing as non-continuous functions can be approximated by continuous ones (basically the entire premise of a digital computer).

I don't think this is right at all. Digital computers express non-continuous functions, and they sometimes use those to approximate continuous functions.

For example, for a function f(x) defined on R with f(x) = -x if x = 0, how would you approximate it by a continuous function g(x) with precision lower than, say, 1 (i.e. |f(x) - g(x)| And of course, there are functions with much worse discontinuities than this.

Re: How deep is the brain? The shallow brain hypothesis

#142

Earlier quoted context omitted.

> matter to construct a physical system following that mathematical model within the bounds, and any such system is equivalent to any other one, within those bounds No. This wasnt discovered. Nearly every physical system is implementing nearly every pure algorithm, ie., every computable function. The particles of gas in the air in my room form a neural network, with the right choice of activation function. Turing-equ…

> Nearly every physical system is implementing nearly every pure algorithm, ie., every computable function. Sure. And also about the air and neural network. This is all irrelevant, for the same reason that every possible program and every possible copyrighted work being contained in the base-10 expansion of the number PI is irrelevant. Or that a photo of every event that ever happened anywhere is contained in the spa…

I think you aren't following the defintion of 'computer' or 'computable', you seem to have a mixed physical notion of what a 'computer' is.

A computer, from a formal pov, is just an abstract mathematical object (like a shape) which has abstract properties (eg., like being a circle) that are computable, ie., are functions from integers to integers.

The physical devices we call 'computers', in many ways, arent. They exist in space and time and hence have non-computable properties, like their (continuous) extension in space and time.

See Turing's own paper where he makes this point himself, ie., that physical machines arent computers in his sense because they're continuous in time.

Insofar as you appeal to any causal aspects of a physical system you arent talking about a computer in turing's sense, and nothing like a turing equivalence would apply.

We already know that all computable functions can be implemented by 'arbitary substrates' -- this is just the same as saying that you can 'make a circle out of any material'.

In exactly the same sense as gears can be networked, sand dunes already are. You can just go around labelling particles of sand with 0s and 1s, and for a subset, there you have it: the computable aspects of the TCP/IP protocol.

But this is irrelevant. TCP/IP isnt useful because of its computable aspects. It's useful as a design sheet for humans to rig systems of electrical devices with highly specific causal properties.

The system we call 'the internet' is useful because it connects keyboards, screens, mice, microphones, webcams, SSDs, RAM, etc. together -- and because these devices are provide for human interaction.

The sand dune is likewise already implementing arbitary computable functions, so is the sun, so is the air, and any arbitary part of the universe you care to choose.

But the sand dune lacks all the properties the internet has: there's no webcam, no keybaord, no screen, etc.

What we actually use are physical properties. Talk of algorithms is just a design tool for people to build stuff

Re: How deep is the brain? The shallow brain hypothesis

#143
post #99

Earlier quoted context omitted.

> Many organisms have just a handful of neurons yet exhibit complex behavior that would be impossible given the weighted connections model. That's rather a bold claim given that artificial neural networks are universal function approximators.

It's incredible to me how widely this is misunderstood. The universal function approximator theorem only applies for continuous functions. Non-continuous functions can only be approximated to the extent that they are of the same "class" as the activation function. Additionally, the theorem only proves that for any given continuous function, there exists a particular NN with particular weight that can approximate that…

> The universal function approximator theorem only applies for continuous functions. Non-continuous functions can only be approximated to the extent that they are of the same "class" as the activation function.

Yes, and?

> Training is not necessarily possible

That would be surprising, do you have any examples?

> and the same NN isn't guaranteed to approximate any other function to some desired precision.

Well duh. Me speaking English doesn't mean I can tell 你好[0] from 泥壕[1] when spoken.

> It seems pretty obvious to me that most interesting behaviours in the real world can't be modelled by a mathematical function at all (that is, for each input having a single output)

I think all of physics would disagree with you there, what with it being built up from functions where each input has a single output. Even Heisenberg uncertainty and quantised results from the Stern-Gerlach setup can be modelled that way in silico to high correspondence with reality, despite the result of testing the Bell inequality meaning there can't be a hidden variable.

[0] Nǐ hǎo, meaning "hello"

[1] Ní háo, which google says is "mud trench", but I wouldn't know

Re: How deep is the brain? The shallow brain hypothesis

#144
post #32

The brain communicates with itself, so deep layers are equivalent to sections of the brain talking to each other. The only relevance white matter depth has is with regard to how it's trained, and since it doesn't use gradient descent, it's irrelevant to neural networks in that regard.

Intercommunication does not equal layer depth.

Why not? All a deep neural network is doing is progressive data transformations into something more abstract and meaningful to later layers.

Re: How deep is the brain? The shallow brain hypothesis

#145

Earlier quoted context omitted.

I think we agree? I am talking to the efficiency of the brain. Not processing speed. Efficiency of the brain to do things advantageous to the selfish genes I guess. The brain is supremely efficient at what the brain has evolved to do. It is almost tautological! Because if it wasn't, it wouldn't have evolved to that. Silicon comes from an alien land, and is emulating. Even with the best algorithms there has to be a li…

> The brain is supremely efficient at what the brain has evolved to do. It is almost tautological! Because if it wasn't, it wouldn't have evolved to that. Not really, evolution doesn't guarantee the brain will be supremely efficient. It just guarantees that it will be efficient ENOUGH.

Again, it is efficient at what it does.

Re: How deep is the brain? The shallow brain hypothesis

#146

Earlier quoted context omitted.

Wrong. Consider two hypothetical versions of this. One, the exact scenario as you described - history unfolded like it did, until the 1900 alien incident. CS and information theory is in its infancy. You're correct that most of the necessary work would first go to physics and chemistry and their various spin-off fields, because that's what's needed to build tools necessary to inspect the machine in full detail. The m…

Nearly every physical system implements every algorithm -- if you wanted to find what in a laptop was 'sorting numbers' that would every part. The light emitted by the screen is being 'sorted' as it is scanned out, the heat air by the fan is being 'sorted' as it swirls around, etc. You cannot ask, "what physical system implements this algorithm?" as an investigative question, the answer is: nearly all of them. This i…

You're normally a lot more coherent than you have been in this thread, so… are you feeling alright? Getting enough sleep?

> The light emitted by the screen is being 'sorted' as it is scanned out, the heat air by the fan is being 'sorted' as it swirls around, etc.

This reads like either you're trolling, or that was written by an LLM, or English isn't your native language, or don't know what 'sorting' is, or you don't know what screens and fans do.

It's so fundamentally wrong I was actually tempted to get ChatGPT to respond to it, but that would be a bit mean and add little.

You're better than this. What's wrong?

Re: How deep is the brain? The shallow brain hypothesis

#147
post #115

I seem to remember research stating that an individual neuron has very complex behaviour that requires several ML “neurons” / nodes to simulate. So if you do a comparison, perhaps the brain is deeper than you’d think by just looking at the graph of neurons and their synapses. Could we construct a neutral net from nodes with more complex behaviour? Probably, but in computing we’ve generally found that it’s best to bui…

> Could we construct a neutral net from nodes with more complex behaviour?

Well there's spiking neural networks (SNN)[1], which are modeled more closely to how neurons actually work.

Main obstacle is still, as far as I know, that there's no way to train a SSN as efficiently as a "regular" neural network, which lends itself very nicely to gradient descent and similar[2].

[1]: https://en.wikipedia.org/wiki/Spiking_neural_network

[2]: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9313413/

Re: How deep is the brain? The shallow brain hypothesis

#148
post #146

Earlier quoted context omitted.

Nearly every physical system implements every algorithm -- if you wanted to find what in a laptop was 'sorting numbers' that would every part. The light emitted by the screen is being 'sorted' as it is scanned out, the heat air by the fan is being 'sorted' as it swirls around, etc. You cannot ask, "what physical system implements this algorithm?" as an investigative question, the answer is: nearly all of them. This i…

You're normally a lot more coherent than you have been in this thread, so… are you feeling alright? Getting enough sleep? > The light emitted by the screen is being 'sorted' as it is scanned out, the heat air by the fan is being 'sorted' as it swirls around, etc. This reads like either you're trolling, or that was written by an LLM, or English isn't your native language, or don't know what 'sorting' is, or you don't…

there's nothing garbled about this idea -- not sure about my messaging in this thread, maybe the explanations are a bit looser today

A computable function is a function from naturals to the naturals typically specified as an algorithm: a sequence of steps by which input numbers are transformed into output numbers.

Eg., consider sorting: 101, 001, 111, etc.

Now any physical system can have any component part associated with 0 or 1. There is no reason, a priori, to suppose that voltage flux on a CPU is a "1" or a "0" any more than to associate a photon emission.

If one associates a photon emission at some location with a 0, and another with a 1, then displaying content on a screen is a form of sorting.

Likewise a planet orbiting the sun is implementing a while(true) i = -1*i, if one associates -1/1 with position of the planet orbiting the sun. This is the heart of 'reversible computing'.

The only reason we associate some microscopic part of a CPU with 0, 1, etc. is by design it is something we as observers bring to bare on our interpretation of the physical system. But there's an infinite number of such attributions. We would only ever come to conclude that voltage flux across transitiors was relevant to the operation of a laptop via physics experiments --- no hope via computer science.

This is very important for understanding why csci is presently useless and misinformative as far as the brain is concerned. There are an infinite number of 0/1 attributions to make, and infinite number of algorithms being implemented etc. almost all of those are irrelevant.

Just, as you detect the absurdity, of using sorting algorithms to understand how an LCD works. This is presently less absurd than people talking about neural networks and equivocating with brain structures

Re: How deep is the brain? The shallow brain hypothesis

#149
post #143

Earlier quoted context omitted.

It's incredible to me how widely this is misunderstood. The universal function approximator theorem only applies for continuous functions. Non-continuous functions can only be approximated to the extent that they are of the same "class" as the activation function. Additionally, the theorem only proves that for any given continuous function, there exists a particular NN with particular weight that can approximate that…

> The universal function approximator theorem only applies for continuous functions. Non-continuous functions can only be approximated to the extent that they are of the same "class" as the activation function. Yes, and? > Training is not necessarily possible That would be surprising, do you have any examples? > and the same NN isn't guaranteed to approximate any other function to some desired precision. Well duh. Me…

> Yes, and?

It means that there is no guarantee that, given a non-continuous function function f(x), there exists an NN that approximates it over its entire domain withing some precision p.

> That would be surprising, do you have any examples?

Do you know of a universal algorithm that can take a continuous function and a target precision, and return an NN architecture (number of layers, number of neurons per layer) and a starting set of weights for an NN, and a training set, such that training the NN will reach the final state?

All I'm claiming is that there is no known algorithm of this kind, and also that the existence of such an algorithm is not guaranteed by any known theorem.

> Well duh. Me speaking English doesn't mean I can tell 你好[0] from 泥壕[1] when spoken.

My point was relevant because we are discussing whether an NN might be equivalent to the human brain, and using the Universal Approximation Theorem to try to decide this. So what I'm saying is that even if "knowning English" were a continuous function and "knowing French" were a continuous function, so by the theorem we know there are NNs that can approximate either one, there is no guarantee that there exists a single NN which can approximate both. There might or might not be one, but the theorem doesn't promise one must exist.

> I think all of physics would disagree with you there, what with it being built up from functions where each input has a single output.

It is built up of them, but there doesn't exist a single function that represents all of physics. You have different functions for different parts of physics. I'm not saying it's not possible a single function could be defined, but I also don't think it's proven that all of physics could be represented by a single function.

Re: How deep is the brain? The shallow brain hypothesis

#150
post #115

I seem to remember research stating that an individual neuron has very complex behaviour that requires several ML “neurons” / nodes to simulate. So if you do a comparison, perhaps the brain is deeper than you’d think by just looking at the graph of neurons and their synapses. Could we construct a neutral net from nodes with more complex behaviour? Probably, but in computing we’ve generally found that it’s best to bui…

> I seem to remember research stating that an individual neuron has very complex behaviour that requires several ML “neurons” / nodes to simulate.

This is probably what you're remembering: https://www.sciencedirect.com/science/article/pii/S089662732...

Post reply on HN