Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching physics to neural networks removes 'chaos blindness'

phys.org

Post reply on HN
Print view View original

41–50 of 81 posts

Re: Teaching physics to neural networks removes 'chaos blindness'

#41

Post by SamReidHughes »

mtgp1000 wrote: post #25
Fricken wrote: post #15

Earlier quoted context omitted.

Don't sell biology short like that. Human driver are born with a mental model of physics that's been honed 24 hours a day since before they were diatoms.

I don't think that's quite right. I believe that humans are essentially born as blank neural networks; it's the structure, and the graph of connections between brain structures and sensory inputs, that is effectively primed for learning certain tasks that we find to be intuitive. A baby is not born with the knowledge of body movement, for example, but through natural exploration of the body and environment, almost al…

I disagree with this, sort of completely, after having been attacked by an animal. There is millions of years of evolution that will wake up and save you from getting hurt as it recognizes what's happening and instantly formulates a response to it.

Re: Teaching physics to neural networks removes 'chaos blindness'

#42

Post by awinter-py »

> the NAIL team incorporated Hamiltonian structure into neural networks

ML non-expert here. Is this the same as having an extra column of your input data that's a hamiltonian of the raw input? Or a kind of neuron that can compute a hamiltonian on an observation? Or something more complicated.

is this like a specialized 'functional region' in a biological brain? (broca's area, cerebellum)

Re: Teaching physics to neural networks removes 'chaos blindness'

#43

Post by rckoepke »

mtgp1000 wrote: post #25
Fricken wrote: post #15

Earlier quoted context omitted.

Don't sell biology short like that. Human driver are born with a mental model of physics that's been honed 24 hours a day since before they were diatoms.

I don't think that's quite right. I believe that humans are essentially born as blank neural networks; it's the structure, and the graph of connections between brain structures and sensory inputs, that is effectively primed for learning certain tasks that we find to be intuitive. A baby is not born with the knowledge of body movement, for example, but through natural exploration of the body and environment, almost al…

https://en.wikipedia.org/wiki/Fixed_action_pattern

More generally, reptiles are born with nearly all the behaviors they'll need throughout life. Why wouldn't humans be born with some?

Re: Teaching physics to neural networks removes 'chaos blindness'

#44

Post by mtgp1000 »

rckoepke wrote: post #43
mtgp1000 wrote: post #25

Earlier quoted context omitted.

I don't think that's quite right. I believe that humans are essentially born as blank neural networks; it's the structure, and the graph of connections between brain structures and sensory inputs, that is effectively primed for learning certain tasks that we find to be intuitive. A baby is not born with the knowledge of body movement, for example, but through natural exploration of the body and environment, almost al…

https://en.wikipedia.org/wiki/Fixed_action_pattern More generally, reptiles are born with nearly all the behaviors they'll need throughout life. Why wouldn't humans be born with some?

>action patterns are said to be produced by the innate releasing mechanism, a "hard-wired" neural network, in response to a sign stimulus or releaser

This is exactly what I'm talking about. Just like a baby deer "instinctively" can walk, but wobbles around for the first few hours, what you're seeing is something very similar to a purpose evolved neural network structure who's weights are being set through the principle of firing and wiring together (I forget what it's called).

I can't believe I got -4 for that!

Edit: hebbian learning. Point is it's probably far too much information to encode in DNA, but if you structure your neural network properly, you encode, how could I put it, the general topology of the problem you are attempting to solve, and through reinforcement learning "fill in the blanks" by training weights (or hebbian learning which functions similarly).

Re: Teaching physics to neural networks removes 'chaos blindness'

#45

Post by vutekst »

awinter-py wrote: post #42

> the NAIL team incorporated Hamiltonian structure into neural networks ML non-expert here. Is this the same as having an extra column of your input data that's a hamiltonian of the raw input? Or a kind of neuron that can compute a hamiltonian on an observation? Or something more complicated. is this like a specialized 'functional region' in a biological brain? (broca's area, cerebellum)

Also ML non-expert here. I think this is about a different kind of neuron(your 2nd suggestion). The paper another commenter linked says:

Hamiltonian neural network (HNN) intakes position and momenta {q,p}, outputs the scalar function H, takes its gradient to find its position and momentum rates of change, and minimizes the loss

which enforces Hamilton's equations of motion.

https://journals.aps.org/pre/abstract/10.1103/PhysRevE.101.0...

Re: Teaching physics to neural networks removes 'chaos blindness'

#46

Post by cmehdy »

fizixer wrote: post #26
cmehdy wrote: post #24

This sounds like the opposite of what Richard Sutton seemed to advocate for in his "Bitter Lesson"[0]. I don't know nearly enough to advocate for one thing or the other, but it is fascinating to see that those approaches seem to compete as we venture into the unknown. [0] http://incompleteideas.net/IncIdeas/BitterLesson.html

They're not the opposite, and both are correct. Sutton is saying 'over a slightly longer time'. You can wait 20 more years and super-duper-deep-NN-on-steroids, and hardware a million times as big and powerful, would rediscover all of theoretical physics. Or you could inject some theoretical physics acquired by humans and make DNNs smarter today.

I assume your 20 years is a guesstimate, and I do think it misses the point of what Sutton's writing is. The trap here is that there's always to be more computing in the future, so where do we draw the line? The idea is to think differently now, for the pursuit of actual progress down the road. Which, by the way, is exactly what people were doing about 40 years ago and what put down more than the foundations for all the tricks we're pulling these days.

Re: Teaching physics to neural networks removes 'chaos blindness'

#47

Post by visarga »

logicslave wrote: post #40
Fricken wrote: post #15

Earlier quoted context omitted.

Don't sell biology short like that. Human driver are born with a mental model of physics that's been honed 24 hours a day since before they were diatoms.

But were all a blank slate!!!

Even blank slates have specific properties, such as being better or worse at handling various types of information.

Re: Teaching physics to neural networks removes 'chaos blindness'

#48

Post by rckoepke »

mtgp1000 wrote: post #44
rckoepke wrote: post #43

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Fixed_action_pattern More generally, reptiles are born with nearly all the behaviors they'll need throughout life. Why wouldn't humans be born with some?

>action patterns are said to be produced by the innate releasing mechanism, a "hard-wired" neural network, in response to a sign stimulus or releaser This is exactly what I'm talking about. Just like a baby deer "instinctively" can walk, but wobbles around for the first few hours, what you're seeing is something very similar to a purpose evolved neural network structure who's weights are being set through the princip…

Your hypothesis feel correct, and is one of two three parts that I feel are missing from current deep learning networks.

1) Pre-existing structures that are already specialized for the necessary tasks, but untrained. We kind of mimic this with transfer learning, and by discovering more appropriate general architectures by hand.

2) Training while inferring. We very crudely approximate this by releasing updated models every month but I think it would be best if also performed at the edge. Google has begun doing this, I have hope for 'federated learning'[0].

3) 20+ years of exaflop training.

More narrowly focused to this article, I believe researchers keep finding that models which are architected to solve the most "general case" possible to solve consistently perform better on highly specific tasks than models trained only on those specific tasks. Definitely creating models that understand general physics follows that trend. Although I suspect, (as I believe you do), that scaling will be hampered without some sort of ML "fixed action patterns".

My thinking about this topic has been strongly guided by a special issue of Scientific American: Mind that I read in 2013 [1]. The issue was hard for me to find today because it's not listed in the usual archives, due to being a special edition. SCIENTIFIC AMERICAN MIND September 2013 Volume 22, Issue 3s

The whole issue is devoted to optical illusions and what they can tell us about how our brain uses evolutionary shortcuts to efficiently determine things in the real world. "In the wild", these shortcuts improve accuracy and speed of inference. But with artificial stimuli, they can lead us astray, and do in the case of artificially generated optical illusions.

As for the -4 (which is the maximum negative you can go on HN) I think some people just saw the first part and clicked downvote at that point.

> I don't think that's quite right. I believe that humans are essentially born as blank neural networks

I wouldn't worry about the vote counter. "Those who play for applause, that's all they'll get." -Wynton Marsalis' dad.

Following up like this to clarify for us idiots is really the best thing to do, maybe editing the original comment for clarity if you really feel like it.

0: https://ai.googleblog.com/2017/04/federated-learning-collabo... 1: https://www.scientificamerican.com/magazine/special-editions...

Re: Teaching physics to neural networks removes 'chaos blindness'

#49

Post by fizixer »

cmehdy wrote: post #46
fizixer wrote: post #26

Earlier quoted context omitted.

They're not the opposite, and both are correct. Sutton is saying 'over a slightly longer time'. You can wait 20 more years and super-duper-deep-NN-on-steroids, and hardware a million times as big and powerful, would rediscover all of theoretical physics. Or you could inject some theoretical physics acquired by humans and make DNNs smarter today.

I assume your 20 years is a guesstimate, and I do think it misses the point of what Sutton's writing is. The trap here is that there's always to be more computing in the future, so where do we draw the line? The idea is to think differently now, for the pursuit of actual progress down the road. Which, by the way, is exactly what people were doing about 40 years ago and what put down more than the foundations for all…

I see what Sutton said as a "statistical learning and artificial intelligence" researcher in line with what the authors of the physics paper presented as "an application of learning research to computational science and engineering, CSE, surrounding physics".

CSE researchers did not sit down and wait for AI researchers to learn the bitter lesson before they resumed their work.

CSE research goes on independent of whether AI/GOFAI/ML has a winter, a summer, an ice age, or a global warming.

It just so happens that in light of the recent progress of AI/ML, specifically 2012 to 2019, they see the utility of incorporating a tiny bit of ML to their vast array of methods.

The paper shared in this thread is merely another attempt to advance such an incorporation. If it doesn't pan out, they go back to doing CSE on physics without any AI or ML.

Post reply on HN