How deep is the brain? The shallow brain hypothesis

And why is this important?

> Do you know of a universal algorithm that can take a continuous function and a target precision, and return an NN architecture (number of layers, number of neurons per layer) and a starting set of weights for an NN, and a training set, such that training the NN will reach the final state?

> All I'm claiming is that there is no known algorithm of this kind, and also that the existence of such an algorithm is not guaranteed by any known theorem.

I think so: the construction proof of the claim that they are universal function approximators seems to meet those requirements.

Even better: it just goes direct to giving you the weights and biases.

> My point was relevant because we are discussing whether an NN might be equivalent to the human brain, and using the Universal Approximation Theorem to try to decide this. So what I'm saying is that even if "knowning English" were a continuous function and "knowing French" were a continuous function, so by the theorem we know there are NNs that can approximate either one, there is no guarantee that there exists a single NN which can approximate both. There might or might not be one, but the theorem doesn't promise one must exist.

I still don't understand your point. It still doesn't seem to matter?

If any organic brain can't do $thing, surely it makes no difference either way whether or not that $thing can or can't be done by whatever function is used by an ANN?

> It is built up of them, but there doesn't exist a single function that represents all of physics. You have different functions for different parts of physics. I'm not saying it's not possible a single function could be defined, but I also don't think it's proven that all of physics could be represented by a single function.

I could point you to this: https://www.youtube.com/watch?v=PHiyQID7SBs

But that would be unfair, given the QM/GR incompatibility.

That said, ultimately I think the onus is on you to demonstrate that it can't be done when all the (known) parts not only already exist separately in such a form, but also, AFAICT, we don't even have a way to describe any possible alternative that wouldn't be made of functions.

I don't think you can say one leg type is better than another without reference to the intended use of the leg - plantigrade legs have better "stability and weight-bearing ability"[0], whereas digitigrade legs (like those of cats and most birds, which BTW appear to have a reverse knee but don't because it is the ankle working like a second backwards knee) "move more quickly and quietly"[1].

Tying this back to the original point, the same is true for brains and computers - they are each better in very specialist cases within specific constraints.

[0] https://en.wikipedia.org/wiki/Plantigrade

[1] https://en.wikipedia.org/wiki/Digitigrade

What makes brain a computer, and the air molecules in your room not a computer, is entropy. The behavior of air molecules is effectively random, the behavior of a brain very much not so.

Also, the universe isn't an uniform temperature soup where everything is equally random. There's energy cost to complexity, and there's a likelihood penalty to complexity. This gives us good confidence that the brain isn't doing something absurdly incomprehensible: it was made by evolution, which is a dumb, brute-force, short-term process. It didn't go out of its way to make things complex - it went with the first random thing that improved survival, which, being random, means generally the simplest thing that could work well enough.

Whatever trickery made brains tick, it must be something that's a) dumb enough for evolution to stumble on it, b) generic enough to scale up by steps small enough for evolution to find, all the way to human level, while c) conferring a survival advantage at every step of the way. Sure, the brain design isn't optimal or made in ways we'd consider elegant, but it's also not actively trying to be confusing. There's literally a survival penalty to being confusing (by means of metabolic cost)!

All to say, we're not dealing with a high-entropy blob of pure randomness. We're dealing with a messy and unusual system, but one that was strongly optimized to be as simple as one could get away with. This narrows down the problem space considerably, and CS is our helpful guide, at the very least by putting lower bounds on complexity of specific computations.