41–50 of 78 posts
This is actually huge. I saw a proof of concept of something like this in Haskell a few years back, but it's amazing it see it (probably) making it into the core of a mainstream language. This may let them capture a large chunk of the ML market from Python - and hopefully greatly improve ML apis while they're at it.
> I saw a proof of concept of something like this in Haskell a few years back, but it's amazing it see it (probably) making it into the core of a mainstream language. Probably Conal Elliott’s work, eg in Vertigo ( http://conal.net/Vertigo/ , circa 2005)? There he was using it for normal computations used in pixel shading, pretty cool stuff. He is still active in this field, and has a lot of new papers that are more M…
Earlier quoted context omitted.
My understanding of automatic differentiation (AD) is that it's only really possible at the compiler level, since you need the ability to interpret and manipulate function definitions themselves. Certainly, no library would be able to offer the same level of guarantees telling you if you've done it wrong, nor the same opportunities for optimisation.
AD is totally possible at the library level (I did it in C# 10 years ago using Conal Elliott’s ideas), however if you want to autodiff more than an expression-only language, compiler support is useful.
I'm not a Swift programmer, so perhaps my confusion is just a symptom of broader ignorance, but I find two things unclear here: - What does 'first-class' mean, really? - Which of these benefits are unique to integrating notions of derivatives into the language, and which could be enjoyed well-written libraries? The mega-proposal links out to a separate doc on embedded DSLs, with broad statements about what's "typical…
From the proposal: > While the differentiation APIs are flexible and fully dynamic, differentiation is based on a program transformation that happens at compile-time. This enables many static analyses that not only help produce more efficient programs, but also detect common numerical programming mistakes such as non-differentiable functions and zero derivatives. > With a first-class differentiable programming langua…
Earlier quoted context omitted.
AD is totally possible at the library level (I did it in C# 10 years ago using Conal Elliott’s ideas), however if you want to autodiff more than an expression-only language, compiler support is useful.
Maybe this is the piece I don't understand. What do you mean by "more than an expression-only language"? What is a non-expression in a language? And what would it mean to have a derivative for your non-expression?
This “trick” does not extend to statements, however. You can’t override if or semicolon in most languages. You can encode statements as expressions, but then you have to worry about things like variable bindings on your own.
Earlier quoted context omitted.
If statements aren't really meaningfully differentiable, regardless of how you do it. Take if x == 59: return 1000 else if x > 59: return -x else: return x How do you optimize this to maximize x, regardless of what language you're in? It's true that you can get a derivative, but the derivative is essentially meaningless.
I don't understand? It's a piecewise differentiable function and you maximize how you maximize any such function: do gradient ascent where it's differentiable and compare against values at the boundary points (ie start, end of interval and points at which there's a removable discontinuity).
Earlier quoted context omitted.
Other people have answered what they do, but this is the big gap between people talking about 'differentiable programming' in theory, and having it actually work in practice. It's true that once you have control flow, the gradient quickly becomes meaningless. I posted an example here: https://news.ycombinator.com/item?id=20892287 That's also the biggest reason I tend to find much of this "differentiable programming"…
> It's hard to reformulate programs in a way s.t. the derivative can mean something meaningful. Really? The gradients computed by AD are the exact answer to the following question: if I were to change this input or parameter an infinitesimal amount, how much would it change the output of my function? That is always meaningful (when it is defined), and means what I just said. You can easily make functions where it is…
For example, take this code.
x,y
for (int i=0; iI believe REINFORCE is not of much help either - it's not magic. I'm not aware of any stochastic gradient estimators that are helpful in this case (although if there is a method I'd like to hear about it).
Earlier quoted context omitted.
If statements aren't really meaningfully differentiable, regardless of how you do it. Take if x == 59: return 1000 else if x > 59: return -x else: return x How do you optimize this to maximize x, regardless of what language you're in? It's true that you can get a derivative, but the derivative is essentially meaningless.
I don't understand? It's a piecewise differentiable function and you maximize how you maximize any such function: do gradient ascent where it's differentiable and compare against values at the boundary points (ie start, end of interval and points at which there's a removable discontinuity).
What I'm arguing is that this gradient will not allow you to optimize anything of interest for the vast majority of programs.
Capitalizing on the presence of people who might be new to Automatic Differentiation and want a deeper understanding of how it works, here is an interactive Colab notebook I wrote about this topic entitled “Build your Own TensorFlow” for the Deep Learning Indaba that just happened in Kenya: https://colab.research.google.com/drive/14GeXkFd5pQKKNIJ7BMs...