How deep is the brain? The shallow brain hypothesis

> And why is this important?

Since we know non-continuous functions are used in describing various physical phenomena, it opens the gate to the possibility that there are physical phenomena that NNs might not be able to learn.

And while piece-wise continuous functions may still be ok, fully discontinuous functions are much harder.

> I think so: the construction proof of the claim that they are universal function approximators seems to meet those requirements.

Oops, you're right, I was too generous. If we know the function, we can easily create the NN, no learning step needed.

The actual challenge I had in mind was to construct an NN for a function which we do not know, but can only sample, such as the "understand English" function. Since we don't know the exact function, we can't use the method from the proof to even construct the network architecture (since we don't know ahead of time how many bumps there are are, we don't know how many hidden neurons to add).

And note that this is an extremely important limitation. After all, if the UAF was good enough, we wouldn't need DL or different network architectures for different domains at all: a single hidden layer is all you need to approximate any continuous function, right?

> If any organic brain can't do $thing, surely it makes no difference either way whether or not that $thing can or can't be done by whatever function is used by an ANN?

Organic brains can obviously learn both English and French. Arguably GPT-4 can too, so maybe this is not the best example.

But the general doubt remains: we know humans express knowledge in a way that doesn't seem contingent upon that knowledge being a single continuous mathematical function. Since the universal function approximator theorem only proves that for each continuous function there exists an NN which approximates it, this theorem doesn't prove that NNs are equivalent to human brains, even in principle.

> That said, ultimately I think the onus is on you to demonstrate that it can't be done when all the (known) parts not only already exist separately in such a form, but also, AFAICT, we don't even have a way to describe any possible alternative that wouldn't be made of functions.

The way physical theories are normally defined is as a set of equations that model a particular process. QM has the Schrodinger equation or its more advanced forms. Classical mechanics has Newton's laws of motion. GR has the Einstein equations. Fluid dynamics has the Navier-Stokes equations. Each of these is defined in terms of mathematical functions: but they are different functions. And yet many humans know all of them.

As we established earlier, the UFA theorem proves that some NN can approximate one function. For 5 functions you can use 5 NNs. But you can't necessarily always combine these into a single NN that can approximate all 5 functions at once. It's trivial if they are simply 5 easily distinguishable inputs which you can combine into a single 5-input function, but not as easy if they are harder to distinguish, or if you don't know that you should model them as different inputs ahead of time.

By the way, there is also an example of a pretty well known mathematical object used in physics that is not actually a proper function - the so-called Dirac delta function. It's not hard to approximate this with an NN at all, but it does show that physics is not strictly speaking limited to functions.

Edit to add: I agree with you that the GP is wrong to claim that the behavior exhibited by some organisms is impossible to explain if we assumed that the brain was equivalent to an (artificial) neural network.

I'm only trying to argue that the reverse is also not proven: that we don't have any proof that an ANN must be equivalent to a human/animal brain in computational power.

Overall, my position is that we just don't know to what extent brains and ANNs correspond to each other.

As soon as you add these physical constraints on what counts as a 'computer' you're no longer talking about computers as specified by turing, nor computer science -- which is better called Discrete Mathematics.

You're conflating the lay sense of the term meaning 'that device that i use' with the technical sense. You cannot attribute properties of one to the other. This is the heart of this AI pseudoscience business.

All circles are topologically equivalent to all squares. That does not mean a square table is 'equivalent' to a circular table in any relevant sense.

If you want to start listing physical constraints: the physical state can be causally set deterministically, the physical state evolves causally, the input and output states are measurable, and so on -- then you end up with a 'physical computer'.

Fine, in doing so you can exclude the air. But you cannot exclude systems incapable of transfering power to devices (ie., useless systems).

So now you add that: a device which, through its operation, powers other devices. You keep doing that and you end up with 'electrical computers' or a very close set of physical objects with physical propeties.

By the time you've enumerated all these physical properties, none of your formal magical 'substrates dont matter' things apply. Indeed, you've just shown how radically the properties of the substrate do apply -- so many properties end up being required.

Now, as far as brains go -- the properties of 'physical computers' do not apply to them: their input/output states may be unmeasurable (eg., if QM is involved); they are not programmable (ie., there is no deterministic way to set their output state); they do not evolve in a causally deterministic way (sensitive to biochemical variation, randomness, etc.).

Either you speak in terms of formalism, in which case you're speaking in applicable non-explanaotry toys of discrete mathematicans'; or you start trying to explain actual physical computers and end up excluding the brain.

All this is to avoid the overwhelmingly obvious point: the study of biological organisms is biology.

> If it was shallow then it wouldn’t take 25 years for a human brain to fully train.

It doesn't. You can speak perfectly fine with children. And in fact some teenagers think they know everything.

Original to Jeff or not, "A Thousand Brains" does a decent job presenting an interesting and highly plausible model of how the neocortext may function.

Your comment would be very valuable to me if it included pointers to better sources. I have sufficient background to see gaps in Jeff's book, and would be interested in exploring these, perhaps through the references you seem to be aware of.

Yeah, biological brains could be remarkably more powerful than digital neural networks if the have primitive functions that we haven't accounted for. For example, some networks seem to encode information in the firing rate, rather than just the presence of a signal. If neurons could, e.g. do frequency-based calculations (and not just threshold-based, like spiking neural nets), they could be orders of magnitude more powerful and efficient. I am thinking particularly about neurons involved in, e.g. audio processing.