Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Momentum Works

distill.pub

Post reply on HN
Print view View original

41–50 of 100 posts

Re: Why Momentum Works

#41

Post by ktta »

codekilla wrote: post #30

It would be really, really great if you could somehow hook this up to Discourse so people could comment on and ask questions about the article. Allowing people to ask questions and having others answer like MathOverflow would I think bring a lot more clarity. Many different kinds of people want to understand material like this but may need the math unpacked in different ways.

I don't see how Discourse will be better than something like HN or reddit. There's also a submission to Reddit[1].

With discourse, I think there will be more noise and a lot of time will be wasted scrolling through unnecessary replies. What's good about the thread-like nature of HN/Reddit is that you'll have proper context and the rating system does its job so everyone's time won't be wasted.

Questions can be answered here and on Reddit too. I think Reddit can be sometimes more helpful than HN when it comes to answering 'easy' questions so I would go there if you have any simple questions.

[1]: https://www.reddit.com/r/MachineLearning/comments/63f3uk/r_w...

Re: Why Momentum Works

#42

Post by panic »

Turn the step size and momentum to maximum for some wonderfully glitchy chaos! (and a demonstration of why forward Euler integration only works well with relatively small step sizes)

Re: Why Momentum Works

#43

Post by soVeryTired »

gabrielgoh wrote: post #36
soVeryTired wrote: post #35

In your polynomial regression example, I can't follow what you mean by p_i = \xi \rightarrow \xi^{i-1} when you're setting up the model.

I just mean p_i(\xi) = \xi^{i-1}. The notation is a little cumbersome here, but it pays of in the second equation

More polynomial regression questions :) Are you using the optimal step size you derived in the previous section? If so, why don't the first and last eigendirections converge at once? If not, doesn't it suggest that there's a trade-off between speed of convergence and the ability to stop early?

In general, this is a nicely written article, though. Good work!

Re: Why Momentum Works

#44

Post by wonderous »

RangerScience wrote: post #23
wonderous wrote: post #17

Reminds me of this paper, "Causal Entropic Forces": http://math.mit.edu/~freer/papers/PhysRevLett_110-168702.pdf

This is one of my favorite things ever. You should also look up "Jeremy England's Entropic Life" for a good companion thought.

A New Physics Theory of Life: https://www.quantamagazine.org/20140122-a-new-physics-theory...

How Life (and Death) Spring From Disorder: https://www.wired.com/2017/02/life-death-spring-disorder/

Re: Why Momentum Works

#45

Post by codekilla »

ktta wrote: post #41
codekilla wrote: post #30

It would be really, really great if you could somehow hook this up to Discourse so people could comment on and ask questions about the article. Allowing people to ask questions and having others answer like MathOverflow would I think bring a lot more clarity. Many different kinds of people want to understand material like this but may need the math unpacked in different ways.

I don't see how Discourse will be better than something like HN or reddit. There's also a submission to Reddit[1]. With discourse, I think there will be more noise and a lot of time will be wasted scrolling through unnecessary replies. What's good about the thread-like nature of HN/Reddit is that you'll have proper context and the rating system does its job so everyone's time won't be wasted. Questions can be answere…

I disagree. The most useful replies can be upvoted to deal with noise, and using mathematical formulas to help explain responses is absolutely crucial. I can't tell you how helpful it has been to be looking at an obscure proof, post a question to Math Overflow, and have the answer explained in an intuitive way with reference to the symbols and notation used.

These articles on distill I believe could greatly benefit from this. Let the community help distill.

Re: Why Momentum Works

#47

Post by xapata »

colah3 wrote: post #9
xapata wrote: post #6

> The added inertia acts both as a smoother and an accelerator momentum = mass * velocity force = mass * acceleration momentum = (force / acceleration) * velocity So, it looks to me like momentum is inversely related to acceleration. It doesn't seem right to call momentum an "accelerator".

Hi! This article is about momentum in the mathematical field of optimization. Acceleration also refers to a phenomenon in optimization. While there are deep connections to their physical analogues, they aren't aren't quite the same thing. If you want to make your analogy work, the momentum algorithm adds mass to an object. In terms of literal acceleration at any point in time it is neutral, but the added mass causes…

The use of the term "accelerator" is misleading as it's introduced during the physical metaphor. You should consider editing that sentence to clarify your meaning:

The added inertia smooths out variation in velocity, dampening oscillations and causing us to barrel through narrow valleys, small humps and local minima. Keeping our speed steadier, we arrive at the global optimum faster.

Note the alliteration :-)

> momentum accelerates convergence

Here it's more clear that momentum is accelerating convergence, not the "heavy ball" itself.

> inertia acts both as a smoother and an accelerator

On the second read, it's more clearly a contradiction. Speed can't be both held steadier and accelerated simultaneously. If you meant that momentum alternately smooths and accelerates, then it's even more strange. For that behavior, some sort of motor would be a more appropriate metaphor.

Re: Why Momentum Works

#48

Post by ktta »

codekilla wrote: post #45
ktta wrote: post #41

Earlier quoted context omitted.

I don't see how Discourse will be better than something like HN or reddit. There's also a submission to Reddit[1]. With discourse, I think there will be more noise and a lot of time will be wasted scrolling through unnecessary replies. What's good about the thread-like nature of HN/Reddit is that you'll have proper context and the rating system does its job so everyone's time won't be wasted. Questions can be answere…

I disagree. The most useful replies can be upvoted to deal with noise, and using mathematical formulas to help explain responses is absolutely crucial. I can't tell you how helpful it has been to be looking at an obscure proof, post a question to Math Overflow, and have the answer explained in an intuitive way with reference to the symbols and notation used. These articles on distill I believe could greatly benefit f…

I see where you're coming from. But maintaining another service and the added expenses, community managers, spam control etc. might be a bit much for something that intended to be a publishing platform.

And if there is a question that requires more control, like math formatting, etc. I would actually suggest posting to cross validated[1] and then linking it here.

[1]: http://stats.stackexchange.com/

Re: Why Momentum Works

#49

Post by gabrielgoh »

soVeryTired wrote: post #43
gabrielgoh wrote: post #36

Earlier quoted context omitted.

I just mean p_i(\xi) = \xi^{i-1}. The notation is a little cumbersome here, but it pays of in the second equation

More polynomial regression questions :) Are you using the optimal step size you derived in the previous section? If so, why don't the first and last eigendirections converge at once? If not, doesn't it suggest that there's a trade-off between speed of convergence and the ability to stop early? In general, this is a nicely written article, though. Good work!

Good question! The parameter has been set to a touch below the optimum. Your observation is accurate, there is indeed such a tradeoff, though it is less than you might think. The qualitative behavior of the system is very sensitive to changes at that point, and the tiny bit of extra convergence you get by getting the step-size exactly right of offset dramatically by the chances of diverging.
Post reply on HN