11–20 of 51 posts
The idea of duality is insane. Its present in linear logic, automatic diff, constructive mathematics, probability, quantum theory, game design, discrete optimization, etc. I wonder if it's related to chirality.
I think the dualities in quantum field theory are different from the dualities in optimization but maybe some category theorist can correct me if I’m wrong
Earlier quoted context omitted.
Category theory isn't quite the correct formalism.
What would be then?
Also alternating graphs or game semantics.
These are all related.
Earlier quoted context omitted.
Discrete optimization and automatic differentiation.
Gimme five and I'll answer two. There's quite a few pairwise permutations and some are easier to understand and more instructive than others. Fundamentally, they are both connected via the idea of convex optimization. Automatic differentiation is a computational technique to solve optimization problems. Yes optimization problems is very general however calculus is a fundamental tool. Dual numbers are somewhat like li…
Earlier quoted context omitted.
They aren't, the underlying idea is the same, or very similar. Pick two of them and I'll provide reference on the connection.
Discrete optimization and automatic differentiation.
Earlier quoted context omitted.
Those are a bunch of different ideas. It's the word "duality" that is insane(ly overloaded).
They aren't, the underlying idea is the same, or very similar. Pick two of them and I'll provide reference on the connection.
Earlier quoted context omitted.
Gimme five and I'll answer two. There's quite a few pairwise permutations and some are easier to understand and more instructive than others. Fundamentally, they are both connected via the idea of convex optimization. Automatic differentiation is a computational technique to solve optimization problems. Yes optimization problems is very general however calculus is a fundamental tool. Dual numbers are somewhat like li…
Curious, can you expand on the connection to convex optimization? To my understanding, discrete optimization is nonconvex by nature due to discontinuities in the feasible space.
Duality is the isomorphism between these two. For example, for humans, it's easier to reason about discrete spaces. However a lot of things simply cannot be done that way.
Think of anything that is tangential (pun intended) to Lie theory. In the context of Lie theory, you have the discrete group and the continuous algebra (the group's tangent space). You go between these two using the exponent (group -> algebra) and logarithm to go back (algebra -> group).
It's the difference between an integral and a Rieman sum. It's the fundamental idea that underlies sampling (say audio sampling or even statistical sampling). You capture some invariants and then you interpolate between these invariants to recreate some smooth curve (or distribution).
The nice thing about the smooth space is that optimization is easy. In the exponential space, addition is multiplication and some expensive things are cheap (computationally speaking).
Earlier quoted context omitted.
For those of us who are not aware of duality in auto differentiation (and haven't had a chance to read the above review), could you introduce the idea? Are you talking about forward mode vs reverse mode -- since I haven't pondered, what's interesting/deep about that?
Edit: I think I answered a different question than I believed. Been a while but my best crack at it: If you’re trying to minimize a function, you can call it the primal function. It will have n inputs and m constraints (like how many of each product should I buy constrained by budget and carrying capacity). You can flip the problem around into its dual formulation. This will be a function with m inputs and n constrai…
Now how does that relate to automatic differentiation I am not sure either.
The idea of duality is insane. Its present in linear logic, automatic diff, constructive mathematics, probability, quantum theory, game design, discrete optimization, etc. I wonder if it's related to chirality.
In other words, lots of things have duals. But the duality between any given pair of things doesn't necessarily expose any deep, fundamental connection to another pair of things which have duality. So it's not that duality features so heavily throughout mathematics as its own concept; rather, we frequently build new theories to tie these things together. It's helpful to be able to translate things from one context to another context.
We could just as easily say that isomorphisms are insane because they feature heavily throughout mathematics. But I don't think that provides a deep insight, because it's not like an isomorphism is a special property that ties a bunch of mathematics together in a grand way. Specific pairs of things can be isomorphic. Likewise specific pairs of things can be duals.
Any given pair of dual things is its own duality. It doesn't necessarily have anything to do with the way another pair of objects is in duality. The terminology here is semantically convenient for intuition, but it's definitely overloaded. I think the commonalities you're seeing here are simply due to the vast utility of linearity in all of those disciplines.
Earlier quoted context omitted.
Discrete optimization and automatic differentiation.
Forward AD is the pushforward of a tangent vector (an element of the tangent space), Reverse AD is a pullback of a cotangent vector (an element of the cotangent space). The duality notion between tangent and cotangent spaces is the same as the duality notion of spaces in optimization. Unfortunately, I'm only passingly familiar with discrete optimization, but I would suspect the notion extends from optimization. That'…
I would agree with the critique: I don't think highlighting duality here is particularly useful. For example, the way dual numbers are used to extend the reals for automatic differentiation doesn't have a deep connection to duality in vector spaces. It's just a very general semantic concept that describes pairs of things. But it doesn't say that any given pair of dual things is related to another pair of dual things.