Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How deep is the brain? The shallow brain hypothesis

nature.com

Post reply on HN
Print view View original

21–30 of 183 posts

Re: How deep is the brain? The shallow brain hypothesis

#21

Post by two_in_one »

jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

While I agree with this emotional post there is one nuance. Neural networks aren't intelligent, brain is. And that's where we want to be. Checking gradients and studying filters can get us only this far. So, using brain as inspiration looks like a good option. There are other, but nobody knows where next breakthrough will be. Like nobody knew five years back that transformers are so powerful. My guess next step to AGI will be a complex modular multi-modal system. With hierarchy, workers and controllers, complex signals.. Sound familiar? Brain is sort of it. This is need for embodied AI, obviously. But, interesting thing, it's needed even for body-less AGI too. I.e. AGI is not a big calculator (!), it's more like real-time system. One reason is that full search is impossible. So, in many cases requests will be like 'give the best answer you can find in 4 seconds'. 'and keep looking'. So far we have only real-time dumb robots and NN big calculators. And brains, of course.

Re: How deep is the brain? The shallow brain hypothesis

#22

Post by edmundsauto »

4death4 wrote: post #8
sheeshkebab wrote: post #2

It’s indeed odd that current dnn’s require massive amount of energy to retrain and lack any kind of practical continuous adaptation and learning.

When you say “massive amount of energy” are you comparing the energy requirements to a single human or to the billions of years of solar and geothermal energy that went into producing the human species?

I don’t think this is an apt comparison, but I do think the amount of energy it takes to grow a human into brain maturity in adulthood is an interesting one. Brains + bodies over a 20 year development cycle is still probably much less than training even a low quality Llm.

Re: How deep is the brain? The shallow brain hypothesis

#23

Post by dilawar »

jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

> previously: comparing brains and "electronic computers")

Before that: comparing brain with hydraulic machines. There has been tendency to compare brain with most complex machine known to us at that particular time.

"Descartes was impressed by the hydraulic figures in the royal gardens, and developed a hydraulic theory of the action of the brain. We have since had telephone theories, electrical field theories, and now theories based on computing machines… . We are more likely to find out how the brain works by studying the brain itself, and the phenomenon of behavior, than by indulging in far-fetched physical analogies." -- Karl Lashley 1951

Re: How deep is the brain? The shallow brain hypothesis

#24

Post by [deleted] »

Affric wrote: post #17
4death4 wrote: post #8

Earlier quoted context omitted.

When you say “massive amount of energy” are you comparing the energy requirements to a single human or to the billions of years of solar and geothermal energy that went into producing the human species?

By that token the amount of energy for neural networks will be bound to some extent by the development of the biosphere and the creators of neural networks.

[deleted]

Re: How deep is the brain? The shallow brain hypothesis

#25

Post by jakobson14 »

jacobsimon wrote: post #19
jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

This is a really weird take. There is such a long history of shared insights between biology and neural network research, and to say they’re unrelated or can’t take inspiration from one another is bizarre. > The concept of CNNs didn't come from biology I just opened a survey paper on CNNs and literally the first sentence of the paper reads: > “Convolutional Neural Network (CNN) is a well-known deep learning architect…

That's later backfill, a retroactive change to give a manufactured "biological" origin story. Whether they're real or not, researchers love a good "we took this from nature, isn't nature wonderful!" explanation.

The C in CNN isn't "Convolution" for no reason. It came from work with convolutional filters (yay Sobel kernels!) which at it's height became filter banks and gabor filters and so on before neural networks pretty much killed off handcrafted feature development. Every explanation of how CNNs work still falls back to the original convolutional kernel intuition.

Re: How deep is the brain? The shallow brain hypothesis

#26

Post by quickthrower2 »

jakobson14 wrote: post #6
quickthrower2 wrote: post #3

Earlier quoted context omitted.

With computer-based intelligence we have the overhead of computing every bit though (probably) inefficient silicon and direct electric currents. The brain leverages the properties of chemicals, though millions of years of evolution.

The brain isn't a faster computer. An infinitely-fast computer wouldn't meaningfully change the "expensive training vs fast, static inference" workflow that neural networks have always been developed around (except in the most brute force-y "retrain on the entire world, every single nanosecond" sense).

I think we agree? I am talking to the efficiency of the brain. Not processing speed. Efficiency of the brain to do things advantageous to the selfish genes I guess.

The brain is supremely efficient at what the brain has evolved to do. It is almost tautological! Because if it wasn't, it wouldn't have evolved to that.

Silicon comes from an alien land, and is emulating. Even with the best algorithms there has to be a limit on how efficient a computer-based intelligence can be without changing how the chips work.

You could spin it around and say, well computers are better at many things than humans, and there is no way you could get a biological brain to be as good for the same amount of power (e.g. a raspberry pi can do calculations our brain couldn't possibly do).

Re: How deep is the brain? The shallow brain hypothesis

#27

Post by wslh »

jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

Only an observer of the topic but I think it is good to review Koch's book about the real complexity of a single neuron [1].

[1] https://www.amazon.com/Biophysics-Computation-Information-Co...

Re: How deep is the brain? The shallow brain hypothesis

#28

Post by drdeca »

jakobson14 wrote: post #18
FeepingCreature wrote: post #16

Earlier quoted context omitted.

First make it work; then make it efficient.

Your scientists were so preoccupied with whether or not they should that they didn't stop to think if they could.

... seems potentially better than the other way around? Well, I suppose it depends.

Re: How deep is the brain? The shallow brain hypothesis

#29

Post by dartos »

jakobson14 wrote: post #25
jacobsimon wrote: post #19

Earlier quoted context omitted.

This is a really weird take. There is such a long history of shared insights between biology and neural network research, and to say they’re unrelated or can’t take inspiration from one another is bizarre. > The concept of CNNs didn't come from biology I just opened a survey paper on CNNs and literally the first sentence of the paper reads: > “Convolutional Neural Network (CNN) is a well-known deep learning architect…

That's later backfill, a retroactive change to give a manufactured "biological" origin story. Whether they're real or not, researchers love a good "we took this from nature, isn't nature wonderful!" explanation. The C in CNN isn't "Convolution" for no reason. It came from work with convolutional filters (yay Sobel kernels!) which at it's height became filter banks and gabor filters and so on before neural networks pr…

You can use that argument for anything you disagree with. Do you have a source or anything?

Re: How deep is the brain? The shallow brain hypothesis

#30

Post by DiggyJohnson »

quickthrower2 wrote: post #26
jakobson14 wrote: post #6

Earlier quoted context omitted.

The brain isn't a faster computer. An infinitely-fast computer wouldn't meaningfully change the "expensive training vs fast, static inference" workflow that neural networks have always been developed around (except in the most brute force-y "retrain on the entire world, every single nanosecond" sense).

I think we agree? I am talking to the efficiency of the brain. Not processing speed. Efficiency of the brain to do things advantageous to the selfish genes I guess. The brain is supremely efficient at what the brain has evolved to do. It is almost tautological! Because if it wasn't, it wouldn't have evolved to that. Silicon comes from an alien land, and is emulating. Even with the best algorithms there has to be a li…

Really well said, I think this is an excellent way to frame the dichotomy (comparison?).

Much of these threads make the binary mistake: can these systems be compared, or are they fundamentally different? A bit of both, almost certainly.

Post reply on HN