Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How deep is the brain? The shallow brain hypothesis

nature.com

Post reply on HN
Print view View original

91–100 of 183 posts

Re: How deep is the brain? The shallow brain hypothesis

#91

Post by mjburgess »

dilawar wrote: post #23
jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

> previously: comparing brains and "electronic computers") Before that: comparing brain with hydraulic machines. There has been tendency to compare brain with most complex machine known to us at that particular time. "Descartes was impressed by the hydraulic figures in the royal gardens, and developed a hydraulic theory of the action of the brain. We have since had telephone theories, electrical field theories, and n…

I cannot agree enough with Karl here. What is the brain? An organic system with deep roots in the organic body, with deep causal connections with its environment.

There's little sense in ignoring the whole basic mode of operation, physics, chemistry and biology of the brain in order to analogise it to another system without any of those properties.

This, at best, provides a set of inspirations for engineers -- it does nothing for science.

Re: How deep is the brain? The shallow brain hypothesis

#92

Post by svara »

jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

Doesn't know what a neurologist is, knows they do shit work.

Re: How deep is the brain? The shallow brain hypothesis

#93

Post by bjornsing »

> This shallow architecture exploits the computational capacity of cortical microcircuits and thalamo-cortical loops that are not included in typical hierarchical deep learning and predictive coding networks.

As I understand it the thalamus is basically a giant switchboard though. I see no reason to believe that it never connects the output of one cortical area to the input of another, thus doubling the effective depth of the neural network. (I haven’t read this paper though, as it was behind a paywall.)

Re: How deep is the brain? The shallow brain hypothesis

#94

Post by PartiallyTyped »

IshKebab wrote: post #87
GranularRecipe wrote: post #86

Earlier quoted context omitted.

The reason for deep learning is that shallow networks are very hard (or impossible) to train. In that sense, long time of training is evidence for shallow networks.

No it's because shallow networks can't express complex functions. If you think about it the shallowest network is pretty much a lookup table. They can theoretically model any function, but the number of parameters needed means in practice they can't. Deep networks can learn much more complex functions for the same number of parameters.

I mean… a 3 layer network is a Universal approximator… and you can very much do network distillation… it’s just that getting them wide enough to learn whatever we want them to isn’t computationally efficient. You end up with much larger matmuls which let’s say for simplicity exhibit cubic scaling in the dim. In contrast, you can stack layers and that comes much more computationally friendly because your matmuls are smaller.

Of course you then need to compensate with residuals, initialisation, normalisation, and all that, but it’s a small price to pay for scaling much much better with compute.

Re: How deep is the brain? The shallow brain hypothesis

#95

Post by anonymousDan »

andbberger wrote: post #60
ramraj07 wrote: post #50

Earlier quoted context omitted.

No one disagrees we might be able to discern insights if we understand how our brain is wired. The problem is the current state of neuroscience is so flawed in its approach it’s not looking like they’re of any use. They don’t even understand how a 900 neuron worms system works but are more than happy to tap half a billion dollars from unsuspecting politicians saying they’ll map the human connectome. Go read the brain…

what are you talking about is this konrad kording's shitposting alt??? this reeks of naivety I certainly have many critiques of methods used in neuroscience rn (as a working neuroscientist) but to reduce those to the conclusion that the entire project of neuroscience is hopeless is absurd. We understand certain things quite well actually, and it's not at all obvious what "understanding" at a larger scale would look l…

It's typical of the arrogant, borderline anti-scientific attitude of a non-negligible fraction of the HN hive mind, i.e. if it came out of academia it must be a waste of time.

Re: How deep is the brain? The shallow brain hypothesis

#96

Post by GranularRecipe »

IshKebab wrote: post #87
GranularRecipe wrote: post #86

Earlier quoted context omitted.

The reason for deep learning is that shallow networks are very hard (or impossible) to train. In that sense, long time of training is evidence for shallow networks.

No it's because shallow networks can't express complex functions. If you think about it the shallowest network is pretty much a lookup table. They can theoretically model any function, but the number of parameters needed means in practice they can't. Deep networks can learn much more complex functions for the same number of parameters.

What the ratio for the number of parameters required to learn some complex function between a shallow network and a deep network (preferably as a function of the complexity)?

Re: How deep is the brain? The shallow brain hypothesis

#97

Post by NoToP »

jakobson14 wrote: post #4

If I had a nickel for every time some neurologist tried to compare brains to neural networks. It's a surefire way to tell someone is either desperate for grant money or has been smoking crack. (previously: comparing brains and "electronic computers") Their entire article hinges on the complaint "brain seems shallow and neural networks are deep, ergo neural networks are doing it wrong." Neurologists seem to have a rea…

I disagree profoundly.

There are things the brain does we have not yet been able to reproduce with a neural network, or to the extent we have seemingly with excessive resources of training and network size. Therefore there is some salient feature of neurology which has been overlooked. I don't think it is necessary to mimic biology down to the exact function of real neurons, but there must in fact be something we are neglecting to mimic.

Re: How deep is the brain? The shallow brain hypothesis

#98

Post by robbrown451 »

krainboltgreene wrote: post #80
robbrown451 wrote: post #35

Earlier quoted context omitted.

I dunno. My comment complained about the parent comment not adding positively to the discussion. And gave at least a bit of support for that complaint. Would you have preferred I emulate your style, and complain while providing no support for my complaint? Ok.

Being positive is not a requirement of commenting on HN, but you should comment with something that is substantive, so yes I do think you shouldn't have commented at all. Tone policing is cringe.

Exactly what are you doing here then?

But hey I guess I can do this too. How's this? Using cringe as an adjective is cringe.

Re: How deep is the brain? The shallow brain hypothesis

#99

Post by ben_w »

aeternum wrote: post #53
p1esk wrote: post #51

Earlier quoted context omitted.

Artificial neural networks are the closest working model of a brain we have today. Lots of graph nodes, with weighted connections, performing distributed computation (mainly hierarchical pattern matching), learning from data by gradually updating weights, using selective attention (and/or recurrence, and/or convolutional filters). Which of the above is not happening in our brains? Which of the above is not biological…

Many organisms have just a handful of neurons yet exhibit complex behavior that would be impossible given the weighted connections model. Not to mention single-celled organisms that exhibit ability to navigate. The model can be the closest working model but that doesn't mean it is complete. It's very likely that cells can store memories/information independent from weights.

> Many organisms have just a handful of neurons yet exhibit complex behavior that would be impossible given the weighted connections model.

That's rather a bold claim given that artificial neural networks are universal function approximators.

Re: How deep is the brain? The shallow brain hypothesis

#100

Post by tudorw »

fastball wrote: post #46
jakobson14 wrote: post #33

Earlier quoted context omitted.

Have a read through the first paper describing a convolutional neural network, from 1998: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf There's absolutely no mention of biological inspiration whatsoever. At the same time, one can point to a long and rich history of convolutional filters being used in signal processing. And then there's the name, Convolutional Neural Network. The entire concept of a CNN is frame…

That is definitely not the first paper describing a CNN. That is not even the first paper by Le Cun describing CNNs (he was already on them as early as 1989[1]). Regardless, Le Cun is not the first to describe CNNs, merely one of the first to use them for OCR (specifically for hand-written text). The first neural network arch to use convolutions instead of matmuls was this[2], from the year of our lord 1988. This in…

Thanks, I was looking for something to do with early work and saccades, didn't find that, but found this;

"The most influential of these early discussions was probably the 1943 paper of Warren McCulloch and Walter Pitts in which activity in neuronal* networks was identified with the operations of the propositional calculus. Actual simulations of recognition automata based on networks were carried out by Frank Rosenblatt before 1958 but the theoretical limitations of his "perceptrons" were soon pointed out by Marvin Minsky and Seymour Papert"

excerpt from a 1998 paper, "Real Brains and Artificial Intelligence" (https://www.jstor.org/stable/20025142)

"Walter Harry Pitts, Jr. (23 April 1923 – 14 May 1969) was an American logician who worked in the field of computational neuroscience.[1]"

'https://en.wikipedia.org/wiki/Walter_Pitts'

Post reply on HN