Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Momentum Works

distill.pub

Post reply on HN
Print view View original

51–60 of 100 posts

Re: Why Momentum Works

#51

Post by ABCLAW »

hardmaru wrote: post #27
mark212 wrote: post #22

I can't follow the math but the presentation is gorgeous (Safari on a MacBook retina display). Really great, keep up the good work!

Which part of the math did you find difficult to follow?

Better question: What background is the reader expected to have?

Until Xi, not a single variable is defined prior to inclusion into an expression. Even Alpha and Beta are only defined in the header diagram rather than the body of the text. Also, why are the iteration notations in the superscript rather than subscript?

And before someone chimes in and says that rudiments are necessary to understand this work, no they aren't. The logical steps here are exceptionally simple (and intuitive - as the introduction might lead you to believe) once you get past the delivery. This is a fantastic article that could become very accessible with the proper notation housekeeping.

Re: Why Momentum Works

#52

Post by stared »

colah3 wrote: post #12
poppingtonic wrote: post #5

I'm really loving the choice of articles, especially since you're just getting started. Edit: I'm referring to the journal, not the author.

Thanks! We're just lucky to have authors like Gabe (@gabrielgoh) come to us with incredible articles. :)

How to know if an article would fit there? For example, I was thinking about adjusting my http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html (already with some interactive components) or writing about RoI polling (https://deepsense.io/region-of-interest-pooling-explained/ - by my colleague, but more interactive).

Would it be on-topic? (After changing style accordingly.)

Re: Why Momentum Works

#53

Post by tzs »

I'm curious about the method chosen to give short term memory to the gradient. The most common way I've seen when people have a time sequence of values X[i] and they want to make a short term memory version Y[i] is to do something of this form:

  Y[i+1] = B * Y[i] + (1-B) * X[i+1]
where 0 Note that if the sequence X becomes a constant after some point, the sequence Y will converge to that constant (as long as B != 1).

For giving the gradient short term memory, the article's approach is of the form:

  Y[i+1] = B * Y[i] + X[i+1]
Note that if X becomes constant, Y converges to X/(1-B), as long as B in [0,1).

Short term memory doesn't really seem to describe what this is doing. There is a memory effect in there, but there is also a multiplier effect when in regions where the input is not changing. So I'm curious how much of the improvement is from the memory effect, and how much from the multiplier effect? Does the more usual approach (the B and 1-B weighting as opposed to a B and 1 weighting) also help with gradient descent?

Re: Why Momentum Works

#56

Post by gabrielgoh »

tzs wrote: post #53

I'm curious about the method chosen to give short term memory to the gradient. The most common way I've seen when people have a time sequence of values X[i] and they want to make a short term memory version Y[i] is to do something of this form: Y[i+1] = B * Y[i] + (1-B) * X[i+1] where 0 Note that if the sequence X becomes a constant after some point, the sequence Y will converge to that constant (as long as B != 1).…

Very good question! I have considered this issue too. This form of weighting is the kind used in ADAM, and is qualitatively different from the updates described here. The tools of analysis in this article can be used to understand that iteration too, (this amounts to a different R matrix) and I would be curious to see if it too allows for a quadratic speedup.

[EDIT] As per halfling's comment, this is just a change of the learning rate by (1-beta)

Re: Why Momentum Works

#57

Post by detaro »

ktta wrote: post #41
codekilla wrote: post #30

It would be really, really great if you could somehow hook this up to Discourse so people could comment on and ask questions about the article. Allowing people to ask questions and having others answer like MathOverflow would I think bring a lot more clarity. Many different kinds of people want to understand material like this but may need the math unpacked in different ways.

I don't see how Discourse will be better than something like HN or reddit. There's also a submission to Reddit[1]. With discourse, I think there will be more noise and a lot of time will be wasted scrolling through unnecessary replies. What's good about the thread-like nature of HN/Reddit is that you'll have proper context and the rating system does its job so everyone's time won't be wasted. Questions can be answere…

HN doesn't work if someone has a question in 2 weeks. Both HN and reddit have an incredible skew towards current things (sort of in the term news aggregator, although current = "whatever was recently submitted", not necessarily = "news"), which doesn't really fit posts like this that are relevant for longer. A subreddit might help, but even there things fall off after a while, regardless of the discussion status.

Re: Why Momentum Works

#58

Post by lutorm »

It would be nice if the introduction made clear that the "Momentum" that "works" is some algorithm and not at all the physical concept "momentum".

Re: Why Momentum Works

#59

Post by halflings »

tzs wrote: post #53

I'm curious about the method chosen to give short term memory to the gradient. The most common way I've seen when people have a time sequence of values X[i] and they want to make a short term memory version Y[i] is to do something of this form: Y[i+1] = B * Y[i] + (1-B) * X[i+1] where 0 Note that if the sequence X becomes a constant after some point, the sequence Y will converge to that constant (as long as B != 1).…

I assume that multiplying by a given factor shouldn't matter since you still have the learning rate as a factor (which is itself a factor of the gradient). This might just mean that the learning rate should be lower or higher with this method.

Re: Why Momentum Works

#60

Post by Animats »

gabrielgoh wrote: post #34
Animats wrote: post #29

Hm. So that helps with high-frequency noise. Any progress on what to do when the dimensions are of vastly different scales? I have an old physics engine which had to solve about 20-value nonlinear differential equations. During a collision, the equations go stiff, and some dimensions may be 10 orders of magnitude steeper than others. Gradient descent then faces very steep knife edges. This is called a "stiff system"…

Author here - I believe the problem of a "stiff system" you're referring to is exactly the problem of pathological curvature! Some points not touched on in the article. If the individual dimensions are of different scales, this problem can be easily fixed with a diagonal preconditioner. Even something like ADAM or Adagrad (unconventional, I know, in this domain) can be used. There's also a small industry around more…

The stiffness may be local. It definitely is in a physical simulation for hard collisions. Machine learning data is usually normalized into [0..1], so if you get a really steep slope, something is pathological.
Post reply on HN