31–40 of 100 posts
For those that don't read materials about optimization as much as they maybe should, what is "w⋆"? It used without introduction and I don't know what it is. Perhaps this a convention I am not aware of?
For those that don't read materials about optimization as much as they maybe should, what is "w⋆"? It used without introduction and I don't know what it is. Perhaps this a convention I am not aware of?
Earlier quoted context omitted.
I'm curious, did the author write the whole article including figures, or did someone else give life to the figures? I can see this type of interactive journal becoming very popular in other fields, but not if the author has to create the diagrams him/herself.
Author here - I've created all the diagrams, though I've received really helpful editorial input from Shan Carter and Chris Olah. If you feel like doing some archeology, you can see for yourself the really ugly drafts in the github history - it isn't pretty! I think these visualizations are deceptively easy to create. Javascript is a powerful language with many libraries, and in my experience, it just took a few nudg…
https://github.com/distillpub/post--momentum/issues/1
We've also had a few designers volunteer to work with researchers on visualization. So, in special cases, we may match-make researchers with designers to produce a great article.
Hm. So that helps with high-frequency noise. Any progress on what to do when the dimensions are of vastly different scales? I have an old physics engine which had to solve about 20-value nonlinear differential equations. During a collision, the equations go stiff, and some dimensions may be 10 orders of magnitude steeper than others. Gradient descent then faces very steep knife edges. This is called a "stiff system"…
Some points not touched on in the article. If the individual dimensions are of different scales, this problem can be easily fixed with a diagonal preconditioner. Even something like ADAM or Adagrad (unconventional, I know, in this domain) can be used.
There's also a small industry around more sophisticated preconditioners for the linear systems in PDEs, see Multigrid, for example, or preconditioned conjugate gradient.
In your polynomial regression example, I can't follow what you mean by p_i = \xi \rightarrow \xi^{i-1} when you're setting up the model.
I can't follow the math but the presentation is gorgeous (Safari on a MacBook retina display). Really great, keep up the good work!
but if you've studied linear systems a bit, the math is laid out really well for understanding and turning into practical applications (which seems to be the whole point of distill.pub).
many math articles, in contrast, seem to obfuscate the "why" and the "what for" and concentrate on the theory and derivation (e.g., many wikipedia math articles). that might be great for mathematical elegance (or cynically, oneupsmanship), but not so great for people trying to build things using that valuable knowledge.
In your polynomial regression example, I can't follow what you mean by p_i = \xi \rightarrow \xi^{i-1} when you're setting up the model.
In your polynomial regression example, I can't follow what you mean by p_i = \xi \rightarrow \xi^{i-1} when you're setting up the model.
I just mean p_i(\xi) = \xi^{i-1}. The notation is a little cumbersome here, but it pays of in the second equation