Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Elements of Differentiable Programming

arxiv.org

Post reply on HN
Print view View original

41–50 of 84 posts

Re: The Elements of Differentiable Programming

#41

Post by MikeBattaglia »

clooper wrote: post #40
MikeBattaglia wrote: post #22

One very interesting thing about automatic differentiation is that you can think of it as involving a new algebra, similar to the complex numbers, where we adjoin an extra element to the reals to form a plane. This new algebra is called the ring of "dual numbers." The difference is that instead of adding a new element "i" with i² = -1, we add one called "h" with h² = 0! Every element in the dual numbers is of the for…

Your are confusing non-standard and dual numbers. The dual numbers are not ordered and contain non-invertible nilpotent elements such as h which squares to 0.

These things are different and I did mean dual numbers. The dual numbers do form an ordered ring. When you complain about certain elements being non-invertible, I think you are probably complaining that they aren't a field.

Re: The Elements of Differentiable Programming

#42

Post by bollu »

fpgamlirfanboy wrote: post #30
MikeBattaglia wrote: post #22

One very interesting thing about automatic differentiation is that you can think of it as involving a new algebra, similar to the complex numbers, where we adjoin an extra element to the reals to form a plane. This new algebra is called the ring of "dual numbers." The difference is that instead of adding a new element "i" with i² = -1, we add one called "h" with h² = 0! Every element in the dual numbers is of the for…

> Most autodiff packages (such as Pytorch) use something not much more advanced than this pytorch absolutely does not use the dual number formulation - there are absolutely no magic epsilons anywhere in pytorch's (or tensorflow's) code base. what you're calling duals are the adjoints where are indeed stored/cached on every node in pytorch graphs. there's a reason no one uses dual numbers (non-standard analysis) for a…

You're right on one count: no library implements forward mode. Hence, you're correct that no autodiff library (including pytorch) implement autodiff this way.

However, *you're wrong* that forward mode cannot be written in terms of dual numbers. The point is that the addition and multiplication operation for dual numbers correspond exactly to rules of the derivative of addition and the derivative of the product.

Re: The Elements of Differentiable Programming

#43

Post by clooper »

MikeBattaglia wrote: post #41
clooper wrote: post #40

Earlier quoted context omitted.

Your are confusing non-standard and dual numbers. The dual numbers are not ordered and contain non-invertible nilpotent elements such as h which squares to 0.

These things are different and I did mean dual numbers. The dual numbers do form an ordered ring. When you complain about certain elements being non-invertible, I think you are probably complaining that they aren't a field.

If a < b then a^2 < b^2. This is not true if you let a = 0 and b = h. The dual numbers do not have an ordering. You should provide sources and proofs next time because it seems like you are just making things up.

Re: The Elements of Differentiable Programming

#44

Post by fpgamlirfanboy »

bollu wrote: post #42
fpgamlirfanboy wrote: post #30

Earlier quoted context omitted.

> Most autodiff packages (such as Pytorch) use something not much more advanced than this pytorch absolutely does not use the dual number formulation - there are absolutely no magic epsilons anywhere in pytorch's (or tensorflow's) code base. what you're calling duals are the adjoints where are indeed stored/cached on every node in pytorch graphs. there's a reason no one uses dual numbers (non-standard analysis) for a…

You're right on one count: no library implements forward mode. Hence, you're correct that no autodiff library (including pytorch) implement autodiff this way. However, *you're wrong* that forward mode cannot be written in terms of dual numbers. The point is that the addition and multiplication operation for dual numbers correspond exactly to rules of the derivative of addition and the derivative of the product.

i didn't say it cannot be, i said there's a natural reason it's not: the same reason the rest of analysis isn't.

> You're right on one count: no library implements forward mode

there are plenty of CFD type libraries that implement forward mode. they also do not use what i'm calling "magical" dual numbers i.e. these nilpotent ring elements.

Re: The Elements of Differentiable Programming

#45

Post by MikeBattaglia »

clooper wrote: post #43
MikeBattaglia wrote: post #41

Earlier quoted context omitted.

These things are different and I did mean dual numbers. The dual numbers do form an ordered ring. When you complain about certain elements being non-invertible, I think you are probably complaining that they aren't a field.

If a < b then a^2 < b^2. This is not true if you let a = 0 and b = h. The dual numbers do not have an ordering. You should provide sources and proofs next time because it seems like you are just making things up.

That isn't what an ordered ring is. Your property of a The correct property is that if a ≤ b, a + c ≤ b + c, and if a ≥ 0 and b ≥ 0, then ab ≥ 0. It is fairly easy to see that these properties hold for dual numbers.

Re: The Elements of Differentiable Programming

#46

Post by hyperbovine »

fpgamlirfanboy wrote: post #17
hyperbovine wrote: post #16

Earlier quoted context omitted.

[flagged]

> graduate-level introduction to differentiable programming. go check out any real graduate textbook. what you will find is they all have exercises. > It's a book for ML researchers. I'm excited to read it. Calm down. just because the authors claim something doesn't make it true. i'm not wrong - this is not a good pedagogical resource and i would bet a year of my salary (as an ML researcher) that you will in fact not…

You certainly have the demeanor of a certain type of ML researcher that I have encountered from time to time ...

Re: The Elements of Differentiable Programming

#47

Post by drdeca »

fpgamlirfanboy wrote: post #32
smokel wrote: post #31

Earlier quoted context omitted.

The book referred to in this post has some information about this. The method with "dual numbers" is called "forward-mode automatic differentiation". PyTorch seems to use "reverse-mode automatic differentiation", which does not use dual numbers, but keeps track of the computation graph.

you guys are wrong and spreading blatant misinformation - there is no magic number whose square is 0 but which is itself not zero anywhere in pytorch or tensorflow or any other real DNN framework that i'm familiar with. it's all fun and games to participate in math woo but you shouldn't be proclaiming things you don't actually know on a public forum.

I don't know why you keep calling it "magic". Whether or not pytorch uses them, they aren't magic, neither in the derogatory sense nor in the praise sense.

Re: The Elements of Differentiable Programming

#48

Post by fpgamlirfanboy »

drdeca wrote: post #47
fpgamlirfanboy wrote: post #32

Earlier quoted context omitted.

you guys are wrong and spreading blatant misinformation - there is no magic number whose square is 0 but which is itself not zero anywhere in pytorch or tensorflow or any other real DNN framework that i'm familiar with. it's all fun and games to participate in math woo but you shouldn't be proclaiming things you don't actually know on a public forum.

I don't know why you keep calling it "magic". Whether or not pytorch uses them, they aren't magic, neither in the derogatory sense nor in the praise sense.

> I don't know why you keep calling it "magic".

because they have all of the gee-whiz factor of a freshman calc proof of the chain rule that divides and multiplies infinitesmals and absolutely not enough of the substance necessary to prove much more than that. they are absolutely, in the research literature, at best an anachronism (harkening back to leibniz) and at worst a parlor trick.

in literally my first response i provided the most trivial counter-example to the magic of non-standard analysis. no answers (crickets). i surmise this is because the people in here talking it up aren't really serious.

Re: The Elements of Differentiable Programming

#49

Post by clooper »

MikeBattaglia wrote: post #45
clooper wrote: post #43

Earlier quoted context omitted.

If a < b then a^2 < b^2. This is not true if you let a = 0 and b = h. The dual numbers do not have an ordering. You should provide sources and proofs next time because it seems like you are just making things up.

That isn't what an ordered ring is. Your property of a The correct property is that if a ≤ b, a + c ≤ b + c, and if a ≥ 0 and b ≥ 0, then ab ≥ 0. It is fairly easy to see that these properties hold for dual numbers.

In my argument a and b are positive and this is true for all ordered rings but not for the dual numbers as you've defined them. Specify the ordering and you will realize h can not be larger nor smaller than 0 because both cases lead to a contradiction.

In any case, I'm dropping out of this thread.

Re: The Elements of Differentiable Programming

#50

Post by MikeBattaglia »

clooper wrote: post #49
MikeBattaglia wrote: post #45

Earlier quoted context omitted.

That isn't what an ordered ring is. Your property of a The correct property is that if a ≤ b, a + c ≤ b + c, and if a ≥ 0 and b ≥ 0, then ab ≥ 0. It is fairly easy to see that these properties hold for dual numbers.

In my argument a and b are positive and this is true for all ordered rings but not for the dual numbers as you've defined them. Specify the ordering and you will realize h can not be larger nor smaller than 0 because both cases lead to a contradiction. In any case, I'm dropping out of this thread.

It isn't true for all ordered rings, and the dual numbers are in fact a counterexample to the claim that it is true.

Beyond that I'm not sure what to tell you, other than it's fairly easy to see that the dual numbers do satisfy the axioms of an ordered ring that I gave. Here's a large survey of various infinitesimal systems by Philip Ehrlich where he also notes the dual numbers are an ordered ring: https://arxiv.org/pdf/1808.03345.pdf.

Post reply on HN