Why Momentum Works • Hacker News Board

Re: Why Momentum Works

#21

Post by simplynot » Tue, Apr 04, 2017, 6:52 PM UTC

colah3 wrote: post #4

simplynot wrote: post #2

But simply not on Firefox

Thanks for pointing that out! We fixed the diagram bug in firefox. There's still bad performance for ~30s after page load -- we're looking into why that's happening -- but after that the page seems to work well.

Cool that you fixed it already! I should have started with some praise: great selection of articles

Re: Why Momentum Works

#22

Post by mark212 » Tue, Apr 04, 2017, 6:53 PM UTC

I can't follow the math but the presentation is gorgeous (Safari on a MacBook retina display). Really great, keep up the good work!

Re: Why Momentum Works

#23

Post by RangerScience » Tue, Apr 04, 2017, 7:07 PM UTC

wonderous wrote: post #17

Reminds me of this paper, "Causal Entropic Forces": http://math.mit.edu/~freer/papers/PhysRevLett_110-168702.pdf

This is one of my favorite things ever. You should also look up "Jeremy England's Entropic Life" for a good companion thought.

Re: Why Momentum Works

#24

Post by Paul-ish » Tue, Apr 04, 2017, 7:19 PM UTC

For those that don't read materials about optimization as much as they maybe should, what is "w⋆"? It used without introduction and I don't know what it is. Perhaps this a convention I am not aware of?

Re: Why Momentum Works

#25

Post by blt » Tue, Apr 04, 2017, 7:23 PM UTC

Paul-ish wrote: post #24

For those that don't read materials about optimization as much as they maybe should, what is "w⋆"? It used without introduction and I don't know what it is. Perhaps this a convention I am not aware of?

Optimal w. It is a convention but still ought to be introduced.

Re: Why Momentum Works

#26

Post by foota » Tue, Apr 04, 2017, 7:43 PM UTC

Paul-ish wrote: post #24

For those that don't read materials about optimization as much as they maybe should, what is "w⋆"? It used without introduction and I don't know what it is. Perhaps this a convention I am not aware of?

In general, something star is the optimal value.

Re: Why Momentum Works

#27

Post by hardmaru » Tue, Apr 04, 2017, 7:48 PM UTC

mark212 wrote: post #22

I can't follow the math but the presentation is gorgeous (Safari on a MacBook retina display). Really great, keep up the good work!

Which part of the math did you find difficult to follow?

Re: Why Momentum Works

#28

Post by gabrielgoh » Tue, Apr 04, 2017, 7:58 PM UTC

brut wrote: post #14

colah3 wrote: post #12

Earlier quoted context omitted.

Thanks! We're just lucky to have authors like Gabe (@gabrielgoh) come to us with incredible articles. :)

I'm curious, did the author write the whole article including figures, or did someone else give life to the figures? I can see this type of interactive journal becoming very popular in other fields, but not if the author has to create the diagrams him/herself.

Author here - I've created all the diagrams, though I've received really helpful editorial input from Shan Carter and Chris Olah. If you feel like doing some archeology, you can see for yourself the really ugly drafts in the github history - it isn't pretty!

I think these visualizations are deceptively easy to create. Javascript is a powerful language with many libraries, and in my experience, it just took a few nudges at exactly the right spots from Shan to go from an idea in my head to fully fledged diagram in distill. The tricky part has always been figuring out what to visualize, and if you're a researcher with an clever idea for a visualization, I recommend you reach out to the distill team.

Re: Why Momentum Works

#29

Post by Animats » Tue, Apr 04, 2017, 7:58 PM UTC

Hm. So that helps with high-frequency noise. Any progress on what to do when the dimensions are of vastly different scales? I have an old physics engine which had to solve about 20-value nonlinear differential equations. During a collision, the equations go stiff, and some dimensions may be 10 orders of magnitude steeper than others. Gradient descent then faces very steep knife edges. This is called a "stiff system" numerically.

Re: Why Momentum Works

#30

Post by codekilla » Tue, Apr 04, 2017, 7:59 PM UTC

It would be really, really great if you could somehow hook this up to Discourse so people could comment on and ask questions about the article. Allowing people to ask questions and having others answer like MathOverflow would I think bring a lot more clarity. Many different kinds of people want to understand material like this but may need the math unpacked in different ways.