Why Deep Learning Works II: the Renormalization Group • Hacker News Board

Re: Why Deep Learning Works II: the Renormalization Group

#21

Post by charleshmartin » Sun, Jul 05, 2015, 6:10 PM UTC

TravisDick wrote: post #14

milesf wrote: post #6

Okay, I confess. I really didn't understand most of that post. It sounds really smart, but someone will have to vouch that it's legit, because the picture of Kadanoff cuddling Cookie Monster trigged my baloney detector https://charlesmartin14.files.wordpress.com/2015/04/kadanoff...

I don't mean to be super negative, but because of the general tone early in the article and some sloppy notation, I never finished reading. I think the goal of an article like this should be to give a high-level intuitive explanation for some technical result, rather than sounding smart or complicated. First, it is a little weird to me to talk about "old-school ML" as learning maps from inputs to hidden features. Tha…

Thanks for the comments. It is helpful to have others read the blog and make suggestions and ask for clarifications. The motivations here are to (1) summarize and clarify the key ideas of the physics paper that observed this connection, and (2) set the stage for my next blog, where I try to connect Deep Learning to my idea around Spin Funnels.

I will review the comments and think how to update the blog to make it more clear.

Re: Why Deep Learning Works II: the Renormalization Group

#22

Post by paulpauper » Sun, Jul 05, 2015, 6:19 PM UTC

I think this is similar to the scalar theory of the stock market, which uses scale invariant geometric objects to represent stock market emery levels

http://greyenlightenment.com/sornette-vs-taleb-debate/

Sornette’s 2013 TED video, in which he predicts an imminent stock market crash due to some ‘power law’, is also wrong because two years later the stock market has continued to rally.

You write on your blog:

These kinds of crashes are not caused by external events or bad players–they are endemic to all markets and result from the cooperative actions of all participants.

Easier said than done. I don't think the log periodic theory is a holy grail to making money in the market. There are too many instances here it has failed, but you cherry-picked a single example with bitcoin where it could have worked.

Re: Why Deep Learning Works II: the Renormalization Group

#23

Post by charleshmartin » Sun, Jul 05, 2015, 6:21 PM UTC

sieisteinmodel wrote: post #19

TravisDick wrote: post #14

Earlier quoted context omitted.

I don't mean to be super negative, but because of the general tone early in the article and some sloppy notation, I never finished reading. I think the goal of an article like this should be to give a high-level intuitive explanation for some technical result, rather than sounding smart or complicated. First, it is a little weird to me to talk about "old-school ML" as learning maps from inputs to hidden features. Tha…

Well, there is more. E.g. abbreviating deep belief nets with DBM, which is the commonly used acronym for deep boltzmann machines. These are similar, but very different. Calling an RBM an encoder is somehow not far fetched, but there are many differences between auto encoders and RBMs. He eventually claims an RBM minimises reconstruction error, which is just plain wrong and shows that this guy has absolutely no clue w…

'Technically' this is correct--the RBM CD algo is not minimizing this function; that's not the point.

It is known that when training an RBM, the reconstruction error decreases but not monotonically; in fact it fluctuates. In the words of Hinton, 'trust it but don't use it'.

http://www.cs.toronto.edu/~hinton/absps/guideTR.pdf (which is cited in the post as well)

So in a global sense, yes, I would say that the RBM does eventually minimize the reconstruction error even though it fluctuates.

I can even offer a conjecture here on why the error fluctuates ; in a discrete RG flow map, there could be finite size effects that would give log-periodic fluctuations. This is a stretch--but it is something that could be tested.

I explain this idea here http://charlesmartin14.wordpress.com/2015/01/16/the-bitcoin-...

As to stacking the RBMs to form a DBN--yeah that's the point. "Hinton showed that RBMs can be stacked and trained in a greedy manner to form so-called Deep Belief Networks (DBN)" http://deeplearning.net/tutorial/DBN.html

Re: Why Deep Learning Works II: the Renormalization Group

#24

Post by noobermin » Sun, Jul 05, 2015, 6:24 PM UTC

And here I thought the renormalization group had no application outside high energy physics and condensed matter. May be I should have stuck with HEP after all.

Re: Why Deep Learning Works II: the Renormalization Group

#25

Post by Animats » Sun, Jul 05, 2015, 6:31 PM UTC

milesf wrote: post #6

Okay, I confess. I really didn't understand most of that post. It sounds really smart, but someone will have to vouch that it's legit, because the picture of Kadanoff cuddling Cookie Monster trigged my baloney detector https://charlesmartin14.files.wordpress.com/2015/04/kadanoff...

I got that feeling too, but I don't understand the subject well enough.

One key concept here is that you want your learning system to have some "free energy" metric which decreases as you run the training set through again and again. This insures some kind of convergence, rather than just thrashing around. Of course, the other problem is getting stuck at a local minimum, which is why you don't want to converge too fast. (I'm not up to speed on that; when I studied AI years ago, everybody was getting stuck at local minima. Now, that's less of a problem, and some of the old algorithms, run with slow learning rates over and over, get stuck less.)

The connection to physics seems to be from conservation of energy. The total energy of a system must decrease. For complex systems where energy is sloshing around from one form to another (flow, turbulence, gas compression, heat, maybe combustion) an energy-based analysis is a way of looking at the problem in a simpler way. That is apparently a useful way to look at deep learning systems. I think this is where the author is coming from.

Whether the connection to physical systems is useful, or merely an interesting analogy, isn't clear from the paper. It may be too soon to tell.

Re: Why Deep Learning Works II: the Renormalization Group

#26

Post by charleshmartin » Sun, Jul 05, 2015, 6:33 PM UTC

paulpauper wrote: post #22

I think this is similar to the scalar theory of the stock market, which uses scale invariant geometric objects to represent stock market emery levels http://greyenlightenment.com/sornette-vs-taleb-debate/ Sornette’s 2013 TED video, in which he predicts an imminent stock market crash due to some ‘power law’, is also wrong because two years later the stock market has continued to rally. You write on your blog: These ki…

It is easier to apply the Sornette theory to antibubbles.

Bitcoin seemed like a great example.

I gotta go back and see how well the predictions actually worked.

Re: Why Deep Learning Works II: the Renormalization Group

#27

Post by contravariant » Sun, Jul 05, 2015, 6:52 PM UTC

oneloop wrote: post #7

milesf wrote: post #6

Okay, I confess. I really didn't understand most of that post. It sounds really smart, but someone will have to vouch that it's legit, because the picture of Kadanoff cuddling Cookie Monster trigged my baloney detector https://charlesmartin14.files.wordpress.com/2015/04/kadanoff...

I will read the post later, but I can tell you right now that I'm a physicist and when I was first introduced to deep learning my first thought, especially in the context of visual recognition, was "this smells of renormalization group".

Do you happen to know any good introductory texts to the renormalization group?

Re: Why Deep Learning Works II: the Renormalization Group

#28

Post by charleshmartin » Sun, Jul 05, 2015, 7:12 PM UTC

albertzeyer wrote: post #2

Note: This is about unsupervised learning and mostly about RBMs/DBNs. Most of the Deep Learning success is all about supervised learning. In the past, RBMs have been used for unsupervised pretraining of the model, however, nowadays, everyone uses supervised pretraining. And the famous DeepMind works (Atari games etc) is mostly about Reinforcement learning, which is again different.

I will address the supervised vs unsupervised issue in my next post. Here, I believe the analogy would be that when a field is applied to a spin glass, it does not exhibit a glass transition to a non-self-averaging (highly non-convex) ground state.

As to supervised vs reinforcement learning, its not that different. See how Vowpal Wabbit incoporates both the 2 ideas in how the SGD update is formulated.

Re: Why Deep Learning Works II: the Renormalization Group

#29

Post by curiousjorge » Sun, Jul 05, 2015, 7:42 PM UTC

It always depresses me when I read anything with math formulas and esoteric terms, a constant reminder of my lifelong incompetence with math and university calculus courses.

Re: Why Deep Learning Works II: the Renormalization Group

#30

Post by sgt101 » Sun, Jul 05, 2015, 9:11 PM UTC

I don't like the assertion at all because so many techniques are held to be "deep learning" and because even when specific techniques are built on an analogy of this sort (think Simulated Annealing and Genetic Algorithms) they do not work "because" they are "like" the physical processes that served as an inspiration.

Names are useful, but only as a aide to thinking. Does this help us think about these techniques?