Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]

Post by wenc » Fri, Mar 08, 2019, 3:15 AM UTC

adamnemecek wrote: post #36

wenc wrote: post #25

Earlier quoted context omitted.

I’m not sure. I’m not entirely convinced that discrete and continuous spaces are dual spaces. They are connected, but they are not duals. Same with sampling vs continuous. One cannot interchange the order of the composing morphisms while preserving the properties of the original. The sampled object cannot reconstruct the continuous object in all situations due to effects like aliasing. In optimization, the concept of…

Look into Chu spaces. > One cannot interchange the order of the composing morphisms while preserving the properties of the original. Good observation one really can't but that was never a hard requirement, right? Ordering becomes actually more interesting because you can have interesting properties like anti-commutativity ( https://en.wikipedia.org/wiki/Anticommutativity ) which is a lot more useful than commutativit…

Post by KenoFischer » Fri, Mar 08, 2019, 3:43 AM UTC

throwawaymath wrote: post #20

KenoFischer wrote: post #15

Earlier quoted context omitted.

Forward AD is the pushforward of a tangent vector (an element of the tangent space), Reverse AD is a pullback of a cotangent vector (an element of the cotangent space). The duality notion between tangent and cotangent spaces is the same as the duality notion of spaces in optimization. Unfortunately, I'm only passingly familiar with discrete optimization, but I would suspect the notion extends from optimization. That'…

Yeah, but all you're really describing here is linear algebra. Vector spaces and linearity are a significant part of every single discipline the grandparent commenter mentioned, but they picked out duality. I would agree with the critique: I don't think highlighting duality here is particularly useful. For example, the way dual numbers are used to extend the reals for automatic differentiation doesn't have a deep con…

Post by thearn4 » Fri, Mar 08, 2019, 4:18 AM UTC

cs702 wrote: post #39

I'm trembling with excitement at the prospect that easy-to-use high-performance automatic differentiation looks likely to become a "must have" capability for more and more computer languages. It's going to become easier and easier to specify objective functions and have the computer optimize programs for a wider range of application.

Post by adamnemecek » Fri, Mar 08, 2019, 4:38 AM UTC

wenc wrote: post #46

adamnemecek wrote: post #44

Earlier quoted context omitted.

Unfortunately this is incorrect. Those minima are not the same. Remember that dual points have a real part and a dual part that indicates the rate of change at that point. The real part is the same but the dual is different.

My point is it’s not possible to uniquely reconstruct an arbitrary polynomial by just knowing the extrema because there may be information loss in the general case. I will stop here.

Post by selimthegrim » Fri, Mar 08, 2019, 6:23 AM UTC

adamnemecek wrote: post #5

ssivark wrote: post #3

Earlier quoted context omitted.

For those of us who are not aware of duality in auto differentiation (and haven't had a chance to read the above review), could you introduce the idea? Are you talking about forward mode vs reverse mode -- since I haven't pondered, what's interesting/deep about that?

AD relies on dual numbers. Dual numbers are more suited for doing calculus. Structurally, dual numbers are numbers of the form a + b * e (where e is epsilon s.t. e^2 = 0 but e != 0. Think of it as the imaginary constant but instead of i^2 = -1, you have e^2=0). For example, multiplication of two dual numbers (a + b * e)(c + d * e) = (ac + ad * e + bc * e +bd * e^2). Since e^2 = 0, you end up with (ac + (ad + bc) * e)…