I was fully expecting an article about some braindead product that nobody needs, called Momentum. Imagine my surprise finding physics and a healthily low percentage of BS.
it's not really physics so much as numerical methods.
Why Momentum Works
91–100 of 100 posts
Re: Why Momentum Works
#92I'm curious about the method chosen to give short term memory to the gradient. The most common way I've seen when people have a time sequence of values X[i] and they want to make a short term memory version Y[i] is to do something of this form: Y[i+1] = B * Y[i] + (1-B) * X[i+1] where 0 Note that if the sequence X becomes a constant after some point, the sequence Y will converge to that constant (as long as B != 1).…
I assume that multiplying by a given factor shouldn't matter since you still have the learning rate as a factor (which is itself a factor of the gradient). This might just mean that the learning rate should be lower or higher with this method.
The question is then really about which method makes it easier to tune parameters or which helps intuition the most.
Re: Why Momentum Works
#93Earlier quoted context omitted.
I'm curious, did the author write the whole article including figures, or did someone else give life to the figures? I can see this type of interactive journal becoming very popular in other fields, but not if the author has to create the diagrams him/herself.
Author here - I've created all the diagrams, though I've received really helpful editorial input from Shan Carter and Chris Olah. If you feel like doing some archeology, you can see for yourself the really ugly drafts in the github history - it isn't pretty! I think these visualizations are deceptively easy to create. Javascript is a powerful language with many libraries, and in my experience, it just took a few nudg…
Thank you very much for the article, I'm really enjoying it. I have a few comments:
- When hovering over some notes I see the citations perfectly, but when there is math involved like for example the one about spectral decay the math is not rendered (I'm using Ubuntu, I have tested both in Firefox and Chrome).
- I see you can open issues on github to send corrections. I wonder if there are other channels of communication also to, for example, address problems like the previous one and also to discuss the article.
- I see people like a lot the figures. I think too they are great but what I really love is the writing and the math exposition.
- I think that a default animation for the figures, if they are meant to be manipulated, would be great.
I will say it again: great article. I have bookmarked the journal and I will proceed to read everything published which looks very promising too.
Re: Why Momentum Works
#94But simply not on Firefox
Thanks for pointing that out! We fixed the diagram bug in firefox. There's still bad performance for ~30s after page load -- we're looking into why that's happening -- but after that the page seems to work well.
The generating of the nice formulas from TeX syntax seems to eat most of those 30s for me - maybe you could do that during the build instead of pushing that work to the client?
Re: Why Momentum Works
#95isn't this akin, in effect, to successive over relaxation? https://en.wikipedia.org/wiki/Successive_over-relaxation under-relax, converge real slowly over-over-relax, oscillate just-right-over-relax, get fast convergence
Its not quite the same. SOR is closer to coordinate descent in the way it acts
Re: Why Momentum Works
#96Re: Why Momentum Works
#97Some of the multi-author articles on Distill have a very important (IMO) innovation. They quantify precisely the contribution each author has made to the article. I would like to see this become norm in scientific papers, so on the one hand it'd be clear who to ask questions if they arise, and on the other the various dignitaries won't get their honorary spot on the authors list of papers they were barely involved wi…
[deleted]
Re: Why Momentum Works
#98I was fully expecting an article about some braindead product that nobody needs, called Momentum. Imagine my surprise finding physics and a healthily low percentage of BS.
This title, and some others on the main page, made me realize that while a title might make sense in the context of its own site, we are collectively really bad at using titles that are appropriate for passing around on the internet. This article (like many others) could have been about just about anything based on the title alone.
Re: Why Momentum Works
#99Earlier quoted context omitted.
But velocity is the integral of acceleration, and for constant acceleration, this means velocity = 1/2*acceleration*time^2 momentum = 1/2*force*time^2 = 1/2*mass*acceleration*time^2 So momentum is actually proportional to acceleration in this case.
Hmm. Think of the response that acceleration would have, all else equal, if momentum increased. There's two ways to change momentum -- velocity or mass. If mass increases, acceleration decreases. If velocity increases, acceleration remains the same. Edit: You've got a bit of an endogeneity problem in your equation. And I guess I do, too. momentum = 1/2 * mass * acceleration * time ^ 2 Since momentum is equivalent to…
'Inertia' works also.
Re: Why Momentum Works
#100Some of the multi-author articles on Distill have a very important (IMO) innovation. They quantify precisely the contribution each author has made to the article. I would like to see this become norm in scientific papers, so on the one hand it'd be clear who to ask questions if they arise, and on the other the various dignitaries won't get their honorary spot on the authors list of papers they were barely involved wi…
As others mentioned, this is common in other fields, it's just not done in machine learning.
We've been reading through the policies of lots of journals that seem thoughtfully run and borrowing good ideas. :P