Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Elements of Differentiable Programming

arxiv.org

Post reply on HN
Print view View original

51–60 of 84 posts

Re: The Elements of Differentiable Programming

#51

Post by bsdpufferfish »

nonagono wrote: post #29
fpgamlirfanboy wrote: post #7

i don't know why people write these things. it's an absolute hodge-podge of theorem/proofs/results/techniques with no unifying theme other than "CALCULUS". so it's a pretty bad math book to actually learn math from (you can always spot a pedagogically unsound math book by its lack of exercises). the book doesn't even have any code in it which is surprising considering it has "programming" in the title. actually i kno…

It’s an introduction to a relatively niche new subfield. If I (an expert in the field but not the subfield) want to learn about differentiable programming, my only option before this monograph was to read through tens of random papers which use different presentation styles, terminology etc. Now I can read through the second half of this, around 100 pages, and jump back to the first half if there’s a prerequisite I d…

Differentiable programming is hardly a "subfield" it can be explained in a paragraph if you know calculus well. If there is any subfield, it's in researching specific compiler optimizations.

Re: The Elements of Differentiable Programming

#52

Post by ubj »

This looks like a great resource! Differentiable programming is such a cool area. I'll never forget writing a differentiable PID controller a few years ago and watching the PID gains get tuned automagically to stabilize the control system. It's powerful stuff if you use it in the right places.

Re: The Elements of Differentiable Programming

#53

Post by kristjansson »

MikeBattaglia wrote: post #22

One very interesting thing about automatic differentiation is that you can think of it as involving a new algebra, similar to the complex numbers, where we adjoin an extra element to the reals to form a plane. This new algebra is called the ring of "dual numbers." The difference is that instead of adding a new element "i" with i² = -1, we add one called "h" with h² = 0! Every element in the dual numbers is of the for…

Dual numbers implement forward mode automatic differentiation, but is there additional value to viewing AD in terms of duals e.g. when we’re implementing reverse mode (backprop)?

Re: The Elements of Differentiable Programming

#54

Post by samatman »

fpgamlirfanboy wrote: post #39
MikeBattaglia wrote: post #38

Earlier quoted context omitted.

Thank you for repeating yourself three times. It seems like you think that the dual number algebra involves "magic woo numbers." It seems like you haven't really worked through this stuff too much. I would suggest reading some of the resources above, such as the MIT lecture series. The rest of your points I think I have already addressed, though you ignored in your reply - I've said Pytorch does reverse mode diff sev…

> It seems like you haven't really worked through this stuff too much yup not at all - i just wandered in off the street and knew accidentally that you were talking about non-standard analysis. > The rest of your points I think I have already addressed please show me the source line number in pytorch or tensorflow that defines this number > we add one called "h" with h² = 0!

You seem somewhat obsessed with the idea that reverse-mode autodiff is not the same technique as forward-mode autodiff. It makes you,,, angry? Seems like such a trivial thing to act a complete fool over.

What's up with that?

Anyway, here's a forward differentiation package with a file that might interest you

https://github.com/JuliaDiff/ForwardDiff.jl/blob/master/src/...

Re: The Elements of Differentiable Programming

#55

Post by fpgamlirfanboy »

samatman wrote: post #54
fpgamlirfanboy wrote: post #39

Earlier quoted context omitted.

> It seems like you haven't really worked through this stuff too much yup not at all - i just wandered in off the street and knew accidentally that you were talking about non-standard analysis. > The rest of your points I think I have already addressed please show me the source line number in pytorch or tensorflow that defines this number > we add one called "h" with h² = 0!

You seem somewhat obsessed with the idea that reverse-mode autodiff is not the same technique as forward-mode autodiff. It makes you,,, angry? Seems like such a trivial thing to act a complete fool over. What's up with that? Anyway, here's a forward differentiation package with a file that might interest you https://github.com/JuliaDiff/ForwardDiff.jl/blob/master/src/...

it's amazing to me that pointing out a straight up mathematically factual inaccuracy is considered "angry" and "acting a fool".

Re: The Elements of Differentiable Programming

#56

Post by nonagono »

bsdpufferfish wrote: post #51
nonagono wrote: post #29

Earlier quoted context omitted.

It’s an introduction to a relatively niche new subfield. If I (an expert in the field but not the subfield) want to learn about differentiable programming, my only option before this monograph was to read through tens of random papers which use different presentation styles, terminology etc. Now I can read through the second half of this, around 100 pages, and jump back to the first half if there’s a prerequisite I d…

Differentiable programming is hardly a "subfield" it can be explained in a paragraph if you know calculus well. If there is any subfield, it's in researching specific compiler optimizations.

Well clearly not, since at least 100 pages of content here are specifically about differentiable programming and not prerequisites :)

More seriously, it's about doing the impossible. Formally, some functions are nondifferentiable, period. But it would be cool if we could actually "more or less" differentiate them. For that we'll necessarily need a bag of tricks which is now coalescing into "techniques" and "principles".

Cf. numerical analysis. It takes a page or two to set up your definitions and show that many functions are badly conditioned, period. And yet we still want to compute them, so we've been building the bag of tricks for almost a century now.

Re: The Elements of Differentiable Programming

#57

Post by hansvm »

fpgamlirfanboy wrote: post #7

i don't know why people write these things. it's an absolute hodge-podge of theorem/proofs/results/techniques with no unifying theme other than "CALCULUS". so it's a pretty bad math book to actually learn math from (you can always spot a pedagogically unsound math book by its lack of exercises). the book doesn't even have any code in it which is surprising considering it has "programming" in the title. actually i kno…

Taste-wise, I'd prefer these sorts of books to just briefly state facts and spend most of the time on building intuition, but just putting it all together in one place is nice in and of itself. I've had to re-derive all this shit on my own the last six months, and it would have been helpful to have a moderately complete compendium of the average practitioner's working knowledge available as a starting reference (admittedly, the first half is a hodge-podge that sort of backgrounds the second). It serves the same purpose as a good "survey" paper.

Taking ordinary calculus knowledge and expanding it to differentiable programming is hard. Even if-else branches have nuances which, at first glance, it looks like the authors didn't handle. They confidently state, for example, that the derivatives of predicate variables are uninformative, but the veracity of that statement depends on the local properties of the branches in question. It's often true for "mathy" conditional statements and often not true for more computer-programming-esque if-statements. Separating a particular representation of a computation from what it actually computes and obtaining accurate partial derivatives is _hard_. Jax and Julia both have bugs on that front, and it's good that people are trying to make this knowledge more accessible (admittedly, not very well on that front IMO, but the other sections I spot-checked all look to be higher quality).

Re: The Elements of Differentiable Programming

#58

Post by samatman »

fpgamlirfanboy wrote: post #55
samatman wrote: post #54

Earlier quoted context omitted.

You seem somewhat obsessed with the idea that reverse-mode autodiff is not the same technique as forward-mode autodiff. It makes you,,, angry? Seems like such a trivial thing to act a complete fool over. What's up with that? Anyway, here's a forward differentiation package with a file that might interest you https://github.com/JuliaDiff/ForwardDiff.jl/blob/master/src/...

it's amazing to me that pointing out a straight up mathematically factual inaccuracy is considered "angry" and "acting a fool".

> there's a reason no one uses dual numbers (non-standard analysis) for anything (neither autodiff nor calculus itself)

wrong

> there are no [dual] numbers anywhere in [...] any other serious autodiff implementation

wrong

> please show me the source line number

did

> i'm wrong and this other guy is right

correct.

> this is why i hate this kind of "TIL, gee whiz" math tidbits

acting a fool. angrily so.

Re: The Elements of Differentiable Programming

#59

Post by bsdpufferfish »

nonagono wrote: post #56
bsdpufferfish wrote: post #51

Earlier quoted context omitted.

Differentiable programming is hardly a "subfield" it can be explained in a paragraph if you know calculus well. If there is any subfield, it's in researching specific compiler optimizations.

Well clearly not, since at least 100 pages of content here are specifically about differentiable programming and not prerequisites :) More seriously, it's about doing the impossible. Formally, some functions are nondifferentiable, period. But it would be cool if we could actually "more or less" differentiate them. For that we'll necessarily need a bag of tricks which is now coalescing into "techniques" and "principle…

> not prerequisites

Almost the entire document is undergrad numerical topics (finite difference methods, maximum likelihood, Jacobins, hessians, newtons method, etc). This is all well covered material that is soundly not research.

Re: The Elements of Differentiable Programming

#60

Post by fpgamlirfanboy »

samatman wrote: post #58
fpgamlirfanboy wrote: post #55

Earlier quoted context omitted.

it's amazing to me that pointing out a straight up mathematically factual inaccuracy is considered "angry" and "acting a fool".

> there's a reason no one uses dual numbers (non-standard analysis) for anything (neither autodiff nor calculus itself) wrong > there are no [dual] numbers anywhere in [...] any other serious autodiff implementation wrong > please show me the source line number did > i'm wrong and this other guy is right correct. > this is why i hate this kind of "TIL, gee whiz" math tidbits acting a fool. angrily so.

is this like some kind of version of truman show? the full sentence is

>please show me the source line number in pytorch or tensorflow that defines this number

why? because the original comment makes a claim about pytorch. it's all right there in black and white.

Post reply on HN