Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How deep is the brain? The shallow brain hypothesis

nature.com

Post reply on HN
Print view View original

131–140 of 183 posts

Re: How deep is the brain? The shallow brain hypothesis

#131

Post by simiones »

IshKebab wrote: post #87
GranularRecipe wrote: post #86

Earlier quoted context omitted.

The reason for deep learning is that shallow networks are very hard (or impossible) to train. In that sense, long time of training is evidence for shallow networks.

No it's because shallow networks can't express complex functions. If you think about it the shallowest network is pretty much a lookup table. They can theoretically model any function, but the number of parameters needed means in practice they can't. Deep networks can learn much more complex functions for the same number of parameters.

> They can theoretically model any function, but the number of parameters needed means in practice they can't.

Even theoretically, no they can't. They can theoretically model any continuos function.

Plus, even for continuous functions, the theorem only proves that, for any function, there exists some NN that approximates it to arbitrary precision. It is not known whether there is some base NN + finite training set that could be used to arrive at that target NN using some algorithm in a finite number of steps.

Re: How deep is the brain? The shallow brain hypothesis

#132

Post by anon291 »

sheeshkebab wrote: post #7
jakobson14 wrote: post #5

Earlier quoted context omitted.

Yes, it's odd that sled dogs make terrible housepets. /s Neural networks fundamentally aren't designed to be otherwise. The workflow that has guided their entire development for over a decade is based around expensive training and static inference.

Why then all the talk about AGI when fundamentals don’t even allow for it to emerge.

Because transformers et al have gotten us the closest we've ever been to any system that can even claim to be AGI.

Re: How deep is the brain? The shallow brain hypothesis

#133

Post by simiones »

ben_w wrote: post #99
aeternum wrote: post #53

Earlier quoted context omitted.

Many organisms have just a handful of neurons yet exhibit complex behavior that would be impossible given the weighted connections model. Not to mention single-celled organisms that exhibit ability to navigate. The model can be the closest working model but that doesn't mean it is complete. It's very likely that cells can store memories/information independent from weights.

> Many organisms have just a handful of neurons yet exhibit complex behavior that would be impossible given the weighted connections model. That's rather a bold claim given that artificial neural networks are universal function approximators.

It's incredible to me how widely this is misunderstood.

The universal function approximator theorem only applies for continuous functions. Non-continuous functions can only be approximated to the extent that they are of the same "class" as the activation function.

Additionally, the theorem only proves that for any given continuous function, there exists a particular NN with particular weight that can approximate that function to a given precision. Training is not necessarily possible, and the same NN isn't guaranteed to approximate any other function to some desired precision.

It seems pretty obvious to me that most interesting behaviors in the real world can't be modelled by a mathematical function at all (that is, for each input having a single output); if we further restrict to continuous functions, or step functions, or whatever restriction we get from our chosen activation function.

Re: How deep is the brain? The shallow brain hypothesis

#134

Post by jacobsimon »

TeMPOraL wrote: post #116
mjburgess wrote: post #90

Earlier quoted context omitted.

Can you explain that reason? Decision trees are called 'trees' for, more or less, the same reason. ie., the diagrammed shape of a decision tree looks a little like the branches of a real one. likewise, in the 50s where diagramming the earliest networks they were aiming to immitate a similar real-world structure. Better that they had called them 'Variable Activation Networks' or some such, and none of this superstitio…

> Better that they had called them 'Variable Activation Networks' or some such But that's the thing: they didn't . Instead, they called them "neural networks". It wasn't random. It feels like part of the field now wants to pretend it was never about how to make a machine think. "No, we're only doing abstract maths, only going on self-contained explorations of CS theory." Yeah, right. That feels like a reaction to the…

Yep! Trees, tree structures, tree diagrams have been regularly in use since the 1700s as a way of defining relationships. https://en.wikipedia.org/wiki/Tree_structure

There’s also a pretty large link between the formal representation of language using syntax trees, which was being formalized by linguists and by programming language developers around the same time: https://en.wikipedia.org/wiki/Formal_language?wprov=sfti1

Re: How deep is the brain? The shallow brain hypothesis

#135

Post by jacobsimon »

ben_w wrote: post #104
jacobsimon wrote: post #79

Earlier quoted context omitted.

I mean, I know you’re just providing an analogy, but people are still studying the physics of bird flight and we’re nowhere close to building machines yet that can maneuver the way birds can. https://www.quantamagazine.org/geometric-analysis-reveals-ho...

I could believe "we have more to learn", but not "we're nowhere close": https://youtu.be/w6VLzKACnS8?si=DZgOPuBRG4Vt98su

TIL

Re: How deep is the brain? The shallow brain hypothesis

#136

Post by bjourne »

civilitty wrote: post #70
andbberger wrote: post #60

Earlier quoted context omitted.

what are you talking about is this konrad kording's shitposting alt??? this reeks of naivety I certainly have many critiques of methods used in neuroscience rn (as a working neuroscientist) but to reduce those to the conclusion that the entire project of neuroscience is hopeless is absurd. We understand certain things quite well actually, and it's not at all obvious what "understanding" at a larger scale would look l…

Agreed. Reading the GP’s comment it feels like it’s from bizzaro world. It’s the computer scientists who have been claiming that neural networks resemble the human brain - they even fucking named them neural networks for christ’s sake! That could be excused as naive hubris in the 1980s, it’s utter delusion now. A surface review of neuroplasticity literature alone should free anyone of the illusion that “neural networ…

Actually it was neuroscientists that developed the models nowadays used for machine learning. The McCulloch-Pitts neuron model introduced in 1943 which lead to Frank Rosenblatt's perceptron introduced in 1958. Machine learning algorithms mostly still use those models but computational neuroscience has progressed towards much more complicated neuronal models.

Re: How deep is the brain? The shallow brain hypothesis

#137

Post by nightski »

simiones wrote: post #131
IshKebab wrote: post #87

Earlier quoted context omitted.

No it's because shallow networks can't express complex functions. If you think about it the shallowest network is pretty much a lookup table. They can theoretically model any function, but the number of parameters needed means in practice they can't. Deep networks can learn much more complex functions for the same number of parameters.

> They can theoretically model any function, but the number of parameters needed means in practice they can't. Even theoretically, no they can't. They can theoretically model any continuos function. Plus, even for continuous functions, the theorem only proves that, for any function, there exists some NN that approximates it to arbitrary precision. It is not known whether there is some base NN + finite training set th…

I'm not sure it is all that interesting of a distinction seeing as non-continuous functions can be approximated by continuous ones (basically the entire premise of a digital computer).

Re: How deep is the brain? The shallow brain hypothesis

#138

Post by lausbub »

chriskanan wrote: post #124

The brain has a lot of skip connections and is massively recurrent. In a sense, the brain can be thought of as having infinite depth due to recurrent thalamno-cortical loops. They do mention thalamno-cortical loops in the paper, so I think a more concrete definition of what is meant by "depth" would be helpful.

The "infinite depth" seems to be a matter of definition. It's practically infinite if you include feedback loops via learning. If you exclude learning, then it's far from "infinite". Activations linger for up to 15-30 seconds, so at oscillations of around 30 Hz that would result in about 450-900 loops (times an unknown small multiplier for the actual number of layers). But the brain presumably only backprops/optimizes a few layers at a time and not much "through" time.

Re: How deep is the brain? The shallow brain hypothesis

#139

Post by mjburgess »

TeMPOraL wrote: post #127
mjburgess wrote: post #123

Earlier quoted context omitted.

To get my point, imagine a laptop was delivered by an alien in the year 1900. Now, try to take that seriously and think about the laptop as an actual object of experimental curiosity -- what exactly does science need to invent, discover, describe etc. to understand the operation of that laptop? 99.999% of that new knowledge has to be in physics and chemistry, before the tiny 0.0001% of theoretical csci knowlegde is b…

Wrong. Consider two hypothetical versions of this. One, the exact scenario as you described - history unfolded like it did, until the 1900 alien incident. CS and information theory is in its infancy. You're correct that most of the necessary work would first go to physics and chemistry and their various spin-off fields, because that's what's needed to build tools necessary to inspect the machine in full detail. The m…

Nearly every physical system implements every algorithm -- if you wanted to find what in a laptop was 'sorting numbers' that would every part.

The light emitted by the screen is being 'sorted' as it is scanned out, the heat air by the fan is being 'sorted' as it swirls around, etc.

You cannot ask, "what physical system implements this algorithm?" as an investigative question, the answer is: nearly all of them.

This is why computable functions, ie., pure algorithms, are explanatorily useless. They play only a (observer-relative) 'design role' in creating real programs.

Re: How deep is the brain? The shallow brain hypothesis

#140

Post by phkahler »

sheeshkebab wrote: post #2

It’s indeed odd that current dnn’s require massive amount of energy to retrain and lack any kind of practical continuous adaptation and learning.

>> It’s indeed odd that current dnn’s require massive amount of energy to retrain and lack any kind of practical continuous adaptation and learning.

To me that just means nobody has figured out how to do that effectively. The majority will simply make use of what's been done and proven, so we got a plateau at object recognition, and again at generative AI (with applications in several domains). One problem with continuous adaptation and learning is providing an "entity" and "environment" for it to "live" in which doing the adaptive learning. There are some researchers doing that either with robots, or simulations. That's much harder to set up than a lot of cloud compute resources. I do agree with you that these aspects are missing and things will be much more interesting when they get addressed.

Post reply on HN