Differentiable Programming Mega-Proposal

Post by chillee » Fri, Sep 06, 2019, 1:43 AM UTC

mlevental wrote: post #27

abeppu wrote: post #20

Earlier quoted context omitted.

It's certainly not the case that autodiff is only possible at the compiler level. I've implemented forward mode (via dual numbers) and reverse mode (via tapes / wengert nodes) autodiff in libraries before.

notice the qualifier "really". obviously you can implement autodiff kind of outside the complier since pytorch and tensor flow exist. but those implementations constrain you to a select few compositions (please no comments on Turing completeness with just loops and conditionals). so for example if statements in pytorch are not differentiable (they might have piece wise continuous derivates) because pytorch doesn't ac…

    if x == 59:
        return 1000
    else if x > 59:
        return -x
    else:
        return x

Post by mlevental » Fri, Sep 06, 2019, 1:52 AM UTC

chillee wrote: post #31

mlevental wrote: post #27

Earlier quoted context omitted.

notice the qualifier "really". obviously you can implement autodiff kind of outside the complier since pytorch and tensor flow exist. but those implementations constrain you to a select few compositions (please no comments on Turing completeness with just loops and conditionals). so for example if statements in pytorch are not differentiable (they might have piece wise continuous derivates) because pytorch doesn't ac…

If statements aren't really meaningfully differentiable, regardless of how you do it. Take if x == 59: return 1000 else if x > 59: return -x else: return x How do you optimize this to maximize x, regardless of what language you're in? It's true that you can get a derivative, but the derivative is essentially meaningless.

Post by taliesinb » Fri, Sep 06, 2019, 2:09 AM UTC

chillee wrote: post #32

wenc wrote: post #13

I'm curious (for folks in the know): how does differentiable programming handle non-differentiable points? Can it detect non-differentiable/non-smooth functions? Non-smooth functions like abs(), max(), min() have points where derivatives do not exist. ReLU functions are non-differentiable at their hinge points. Disjoint IF-THEN-ELSE conditions are discontinuities in the function space, and are traditionally handled i…

Other people have answered what they do, but this is the big gap between people talking about 'differentiable programming' in theory, and having it actually work in practice. It's true that once you have control flow, the gradient quickly becomes meaningless. I posted an example here: https://news.ycombinator.com/item?id=20892287 That's also the biggest reason I tend to find much of this "differentiable programming"…

Post by seanmcdirmid » Fri, Sep 06, 2019, 2:20 AM UTC

thedataangel wrote: post #16

abeppu wrote: post #11

I'm not a Swift programmer, so perhaps my confusion is just a symptom of broader ignorance, but I find two things unclear here: - What does 'first-class' mean, really? - Which of these benefits are unique to integrating notions of derivatives into the language, and which could be enjoyed well-written libraries? The mega-proposal links out to a separate doc on embedded DSLs, with broad statements about what's "typical…

My understanding of automatic differentiation (AD) is that it's only really possible at the compiler level, since you need the ability to interpret and manipulate function definitions themselves. Certainly, no library would be able to offer the same level of guarantees telling you if you've done it wrong, nor the same opportunities for optimisation.

Post by seanmcdirmid » Fri, Sep 06, 2019, 2:25 AM UTC

thedataangel wrote: post #5

This is actually huge. I saw a proof of concept of something like this in Haskell a few years back, but it's amazing it see it (probably) making it into the core of a mainstream language. This may let them capture a large chunk of the ML market from Python - and hopefully greatly improve ML apis while they're at it.