Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
1–10 of 51 posts
Re: Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
#2The idea of duality is insane. Its present in linear logic, automatic diff, constructive mathematics, probability, quantum theory, game design, discrete optimization, etc.
I wonder if it's related to chirality.
Re: Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
#3The idea of duality is insane. Its present in linear logic, automatic diff, constructive mathematics, probability, quantum theory, game design, discrete optimization, etc. I wonder if it's related to chirality.
For those of us who are not aware of duality in auto differentiation (and haven't had a chance to read the above review), could you introduce the idea? Are you talking about forward mode vs reverse mode -- since I haven't pondered, what's interesting/deep about that?
Re: Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
#4The idea of duality is insane. Its present in linear logic, automatic diff, constructive mathematics, probability, quantum theory, game design, discrete optimization, etc. I wonder if it's related to chirality.
I think the dualities in quantum field theory are different from the dualities in optimization but maybe some category theorist can correct me if I’m wrong
Re: Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
#5The idea of duality is insane. Its present in linear logic, automatic diff, constructive mathematics, probability, quantum theory, game design, discrete optimization, etc. I wonder if it's related to chirality.
For those of us who are not aware of duality in auto differentiation (and haven't had a chance to read the above review), could you introduce the idea? Are you talking about forward mode vs reverse mode -- since I haven't pondered, what's interesting/deep about that?
AD relies on dual numbers. Dual numbers are more suited for doing calculus.
Structurally, dual numbers are numbers of the form a + b * e (where e is epsilon s.t. e^2 = 0 but e != 0. Think of it as the imaginary constant but instead of i^2 = -1, you have e^2=0).
For example, multiplication of two dual numbers (a + b * e)(c + d * e) = (ac + ad * e + bc * e +bd * e^2). Since e^2 = 0, you end up with (ac + (ad + bc) * e).
Here comes the magic. Dual numbers let you evaluate a function and get the derivative at that point by just evaluating the function. In the above, example, if this was the result of a function, ac is the value of the function and ad + bc is the derivative of that function at that value.
https://blog.demofox.org/2014/12/30/dual-numbers-automatic-d...
Re: Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
#6The idea of duality is insane. Its present in linear logic, automatic diff, constructive mathematics, probability, quantum theory, game design, discrete optimization, etc. I wonder if it's related to chirality.
For those of us who are not aware of duality in auto differentiation (and haven't had a chance to read the above review), could you introduce the idea? Are you talking about forward mode vs reverse mode -- since I haven't pondered, what's interesting/deep about that?
Edit: I think I answered a different question than I believed.
Been a while but my best crack at it: If you’re trying to minimize a function, you can call it the primal function. It will have n inputs and m constraints (like how many of each product should I buy constrained by budget and carrying capacity). You can flip the problem around into its dual formulation. This will be a function with m inputs and n constraints, and it will be a maximization problem.
If you can solve the dual formulation (global max), you know that the primal cannot possibly be lower than the dual. For some types of problems (convex), you can even guarantee that the global max of the dual is the global min of the primal.
The transformation between primal and dual formulations goes both ways, so the primal is the dual of the dual.
Re: Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
#7The idea of duality is insane. Its present in linear logic, automatic diff, constructive mathematics, probability, quantum theory, game design, discrete optimization, etc. I wonder if it's related to chirality.
Those are a bunch of different ideas. It's the word "duality" that is insane(ly overloaded).
Re: Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
#8The idea of duality is insane. Its present in linear logic, automatic diff, constructive mathematics, probability, quantum theory, game design, discrete optimization, etc. I wonder if it's related to chirality.
Those are a bunch of different ideas. It's the word "duality" that is insane(ly overloaded).
They aren't, the underlying idea is the same, or very similar. Pick two of them and I'll provide reference on the connection.
Re: Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
#9Earlier quoted context omitted.
Those are a bunch of different ideas. It's the word "duality" that is insane(ly overloaded).
They aren't, the underlying idea is the same, or very similar. Pick two of them and I'll provide reference on the connection.
Discrete optimization and automatic differentiation.
Re: Automatic Differentiation in Machine Learning: A Survey (2018) [pdf]
#10Earlier quoted context omitted.
They aren't, the underlying idea is the same, or very similar. Pick two of them and I'll provide reference on the connection.
Discrete optimization and automatic differentiation.
Gimme five and I'll answer two. There's quite a few pairwise permutations and some are easier to understand and more instructive than others.
Fundamentally, they are both connected via the idea of convex optimization. Automatic differentiation is a computational technique to solve optimization problems.
Yes optimization problems is very general however calculus is a fundamental tool. Dual numbers are somewhat like lie groups, very smooth and conducive to optimization.