Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How deep is the brain? The shallow brain hypothesis

nature.com

Post reply on HN
Print view View original

31–40 of 183 posts

Re: How deep is the brain? The shallow brain hypothesis

#31

Post by krainboltgreene »

robbrown451 wrote: post #13
jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

I can't agree with the dismissiveness of this comment, and frankly I find its tone out of line and not with the spirit of Hacker News. There are insights that can come from studying the brain, that do indeed apply. Some researchers may not glean anything from such studies, and some may. I have no doubt that as neural networks get more an more powerful, we will continue to find more ways they are similar to the brain,…

What does this comment add to the discussion?

Re: How deep is the brain? The shallow brain hypothesis

#32

Post by Salgat »

The brain communicates with itself, so deep layers are equivalent to sections of the brain talking to each other. The only relevance white matter depth has is with regard to how it's trained, and since it doesn't use gradient descent, it's irrelevant to neural networks in that regard.

Re: How deep is the brain? The shallow brain hypothesis

#33

Post by jakobson14 »

dartos wrote: post #29
jakobson14 wrote: post #25

Earlier quoted context omitted.

That's later backfill, a retroactive change to give a manufactured "biological" origin story. Whether they're real or not, researchers love a good "we took this from nature, isn't nature wonderful!" explanation. The C in CNN isn't "Convolution" for no reason. It came from work with convolutional filters (yay Sobel kernels!) which at it's height became filter banks and gabor filters and so on before neural networks pr…

You can use that argument for anything you disagree with. Do you have a source or anything?

Have a read through the first paper describing a convolutional neural network, from 1998: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf

There's absolutely no mention of biological inspiration whatsoever. At the same time, one can point to a long and rich history of convolutional filters being used in signal processing. And then there's the name, Convolutional Neural Network. The entire concept of a CNN is framed as a series of learned filters.

Re: How deep is the brain? The shallow brain hypothesis

#34

Post by 4death4 »

edmundsauto wrote: post #22
4death4 wrote: post #8

Earlier quoted context omitted.

When you say “massive amount of energy” are you comparing the energy requirements to a single human or to the billions of years of solar and geothermal energy that went into producing the human species?

I don’t think this is an apt comparison, but I do think the amount of energy it takes to grow a human into brain maturity in adulthood is an interesting one. Brains + bodies over a 20 year development cycle is still probably much less than training even a low quality Llm.

Let’s say a human needs an average of 2000 calories a day. A calorie is roughly equivalent to 1 Watt hour, so over 20 years, it takes about 15 MWh to sustain a human.

Let’s say a single A100 has a peak power draw of 250W, and you need 100 to train an LLM. So each hour of training consumes 25,000 Wh of energy. 15 MWh / 25,000 W = 600 hours, or 25 days, which is probably pretty close to the true training time.

So the numbers are actually pretty close. But a human brain doesn’t start out as a set of random weights like an LLM. The human brain has predefined structure that’s the result of an extremely long evolutionary process.

Re: How deep is the brain? The shallow brain hypothesis

#35

Post by robbrown451 »

krainboltgreene wrote: post #31
robbrown451 wrote: post #13

Earlier quoted context omitted.

I can't agree with the dismissiveness of this comment, and frankly I find its tone out of line and not with the spirit of Hacker News. There are insights that can come from studying the brain, that do indeed apply. Some researchers may not glean anything from such studies, and some may. I have no doubt that as neural networks get more an more powerful, we will continue to find more ways they are similar to the brain,…

What does this comment add to the discussion?

I dunno. My comment complained about the parent comment not adding positively to the discussion. And gave at least a bit of support for that complaint.

Would you have preferred I emulate your style, and complain while providing no support for my complaint?

Ok.

Re: How deep is the brain? The shallow brain hypothesis

#36

Post by SubiculumCode »

jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

If you read this article, I think most would understand that it is primarily aimed at other neuroscientists, and only using ML structures an an analogy only, and I think a somewhat useful one to boot. The real point of the article was to propose a general hierarchy for how information flows in the brain, to emphasize the importance of subcortical brain even in higher order cognition, and proposes how simultaneous processing of multiple levels of representation can inform action and thought.

As a developmental neuroscientist, I found the article insightful and thought provoking. Further, it is quite consistent with major hypotheses in psychology, how the hippocampus works (a subcortical structure) and combines information into memories: See fuzzy trace theory [1], for example.

Your dismissive tone is unappreciated, ill-informed, and crass.

[1] https://en.wikipedia.org/wiki/Fuzzy-trace_theory

Re: How deep is the brain? The shallow brain hypothesis

#37

Post by bjourne »

jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

First, I wonder how you got access to the article? It is behind a paywall and not yet uploaded to the sites I usually find paywalled articles on.

Second, there is no need to compare brains to neural networks because brains are neural networks. Neurons form vertices and axons edges connecting the aforementioned. What you are perhaps thinking of are artificial neural networks - most of which are very dissimilar to brains. But even then you are wrong. Artificial Izhikevich and Hodgkin-Huxley neural networks attempts to closely mimic the behavior of real neurons.

While deep, hierarchical artificial neural networks have been more successful than biologically plausible ones, that may be because the technology isn't ready yet. After all, the perceptron was invented in the 1950's but didn't become prominent until the 2010's (or so). Perhaps we need new memories that better map to (real) neural network topologies, or perhaps 3d chips that can pack transistors in the same way brains pack neurons.

Re: How deep is the brain? The shallow brain hypothesis

#38

Post by readthenotes1 »

jakobson14 wrote: post #33
dartos wrote: post #29

Earlier quoted context omitted.

You can use that argument for anything you disagree with. Do you have a source or anything?

Have a read through the first paper describing a convolutional neural network, from 1998: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf There's absolutely no mention of biological inspiration whatsoever. At the same time, one can point to a long and rich history of convolutional filters being used in signal processing. And then there's the name, Convolutional Neural Network. The entire concept of a CNN is frame…

That's funny. I had a book on "neural nets" in the 1980s, and it mentioned the analog to brain neurons.

Re: How deep is the brain? The shallow brain hypothesis

#39

Post by SubiculumCode »

jakobson14 wrote: post #20
crustacean111 wrote: post #15

Earlier quoted context omitted.

CNNs actually are biologically inspired. The receptive field in a CNN mimics the way that cortical neurons only respond to stimuli in a restricted region of the visual field. Different cortical neurons have receptive fields that partially overlap to cover the whole visual field [1]. [1] - https://en.wikipedia.org/wiki/Convolutional_neural_network

You're going to have to dig deeper. The concept of a receptive field goes all the way back to convolutional filters. It's not surprising that we found out later the brain also uses such a fundamental element of signal theory.

Oh good. So you do admit that there are useful parallels between signal processing, statistical processing, and the brain.

Re: How deep is the brain? The shallow brain hypothesis

#40

Post by __loam »

jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

As a biomedical engineer who went into software, thank you for this comment lol. So tired of rehashing this.
Post reply on HN