Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differentiable Programming Mega-Proposal

forums.swift.org

Post reply on HN
Print view View original

11–20 of 78 posts

Re: Differentiable Programming Mega-Proposal

#11

Post by abeppu »

I'm not a Swift programmer, so perhaps my confusion is just a symptom of broader ignorance, but I find two things unclear here: - What does 'first-class' mean, really? - Which of these benefits are unique to integrating notions of derivatives into the language, and which could be enjoyed well-written libraries?

The mega-proposal links out to a separate doc on embedded DSLs, with broad statements about what's "typical" or "often" true of existing DSLs -- but which of those issues are insurmountable? That section mentions Swift's limited metaprogramming facilities. Why choose to carve out this added support for a single family of algorithms rather than add in some more general metaprogramming abilities that enable better EDSLs?

Re: Differentiable Programming Mega-Proposal

#12

Post by taliesinb »

Capitalizing on the presence of people who might be new to Automatic Differentiation and want a deeper understanding of how it works, here is an interactive Colab notebook I wrote about this topic entitled “Build your Own TensorFlow” for the Deep Learning Indaba that just happened in Kenya: https://colab.research.google.com/drive/14GeXkFd5pQKKNIJ7BMs...

Re: Differentiable Programming Mega-Proposal

#13

Post by wenc »

I'm curious (for folks in the know): how does differentiable programming handle non-differentiable points? Can it detect non-differentiable/non-smooth functions?

Non-smooth functions like abs(), max(), min() have points where derivatives do not exist. ReLU functions are non-differentiable at their hinge points.

Disjoint IF-THEN-ELSE conditions are discontinuities in the function space, and are traditionally handled in optimization with mixed-integer formulations (i.e. split up the space and do something clever like branch-and-bound to find the optimum).

Re: Differentiable Programming Mega-Proposal

#15

Post by krapht »

thedataangel wrote: post #5

This is actually huge. I saw a proof of concept of something like this in Haskell a few years back, but it's amazing it see it (probably) making it into the core of a mainstream language. This may let them capture a large chunk of the ML market from Python - and hopefully greatly improve ML apis while they're at it.

Huh? Nobody is writing numerically intensive libraries in Python. Clearly this language proposal is taking aim at C++ and Fortran. Even if this caused TensorFlow & others to rewrite everything in Swift, people would write Python bindings to it and keep using Python.

I'll get excited if Apple actually merges this into Swift. It's a niche feature that their compiler team will need to maintain forever. I actually have been working on algorithmic differentiation in C++, so it's not even that I wouldn't want to try Swift out if it actually made it in. However, because this sort of thing is of such narrow interest I believe the future will stay with embedded DSLs / libraries / ugly macro/template hackery.

Re: Differentiable Programming Mega-Proposal

#16

Post by thedataangel »

abeppu wrote: post #11

I'm not a Swift programmer, so perhaps my confusion is just a symptom of broader ignorance, but I find two things unclear here: - What does 'first-class' mean, really? - Which of these benefits are unique to integrating notions of derivatives into the language, and which could be enjoyed well-written libraries? The mega-proposal links out to a separate doc on embedded DSLs, with broad statements about what's "typical…

My understanding of automatic differentiation (AD) is that it's only really possible at the compiler level, since you need the ability to interpret and manipulate function definitions themselves. Certainly, no library would be able to offer the same level of guarantees telling you if you've done it wrong, nor the same opportunities for optimisation.

Re: Differentiable Programming Mega-Proposal

#17

Post by abeppu »

wenc wrote: post #13

I'm curious (for folks in the know): how does differentiable programming handle non-differentiable points? Can it detect non-differentiable/non-smooth functions? Non-smooth functions like abs(), max(), min() have points where derivatives do not exist. ReLU functions are non-differentiable at their hinge points. Disjoint IF-THEN-ELSE conditions are discontinuities in the function space, and are traditionally handled i…

My understanding is you don't explicitly handle that. For whichever place you're evaluating, you have some computation graph, and you use the elementary operators on that graph. So, if you have defined ReLU with an `if x >= 0 return x` clause, then if you evaluate the derivative at 0, that's the branch you go down, and you say the answer is 1.

Re: Differentiable Programming Mega-Proposal

#18

Post by taliesinb »

wenc wrote: post #13

I'm curious (for folks in the know): how does differentiable programming handle non-differentiable points? Can it detect non-differentiable/non-smooth functions? Non-smooth functions like abs(), max(), min() have points where derivatives do not exist. ReLU functions are non-differentiable at their hinge points. Disjoint IF-THEN-ELSE conditions are discontinuities in the function space, and are traditionally handled i…

While the gradients don’t technically exist, you usually use the limiting value from one side (e.g for the commonly used ReLU activation, you can use either 0 or 1). This is justified by the observation that randomly initialized networks are extremely unlikely to encounter these particular values without some kinder of deeper conspiracy happening. You could put this on a mathematical footing by saying the set of non differentiable points has measure zero, for example.

Re: Differentiable Programming Mega-Proposal

#19

Post by [deleted] »

wenc wrote: post #13

I'm curious (for folks in the know): how does differentiable programming handle non-differentiable points? Can it detect non-differentiable/non-smooth functions? Non-smooth functions like abs(), max(), min() have points where derivatives do not exist. ReLU functions are non-differentiable at their hinge points. Disjoint IF-THEN-ELSE conditions are discontinuities in the function space, and are traditionally handled i…

[deleted]

Re: Differentiable Programming Mega-Proposal

#20

Post by abeppu »

thedataangel wrote: post #16
abeppu wrote: post #11

I'm not a Swift programmer, so perhaps my confusion is just a symptom of broader ignorance, but I find two things unclear here: - What does 'first-class' mean, really? - Which of these benefits are unique to integrating notions of derivatives into the language, and which could be enjoyed well-written libraries? The mega-proposal links out to a separate doc on embedded DSLs, with broad statements about what's "typical…

My understanding of automatic differentiation (AD) is that it's only really possible at the compiler level, since you need the ability to interpret and manipulate function definitions themselves. Certainly, no library would be able to offer the same level of guarantees telling you if you've done it wrong, nor the same opportunities for optimisation.

It's certainly not the case that autodiff is only possible at the compiler level. I've implemented forward mode (via dual numbers) and reverse mode (via tapes / wengert nodes) autodiff in libraries before.
Post reply on HN