Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Differentiable Programming – A Simple Introduction

assemblyai.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Differentiable Programming – A Simple Introduction

#41

Post by nerdponx »

6gvONxR4sf7o wrote: post #32
potbelly83 wrote: post #15

Earlier quoted context omitted.

How do you differentiate a string? Enum?

The answer to that is a huge part of the NLP field. The current answer is that you break down the string into constituent parts and map each of them into a high dimensional space. “cat” becomes a large vector whose position is continuous and therefore differentiable. “the cat” probably becomes a pair of vectors.

It's weirder than that. You typically are differentiating a loss function of strings and various opaque weights. You are optimizing the loss function over the weight space, so in some informal contrived sense you are actually differentiating with respect to the string.

Re: Differentiable Programming – A Simple Introduction

#42

Post by tome »

orbifold wrote: post #26
tome wrote: post #23

Earlier quoted context omitted.

> the whole structure is easily converted to reverse mode afterwards. Unfortunately it's not. Elliot never actually demonstrates in the paper how to implement such an algorithm, and it's very hard to write compiler transformations in "categorical form". (Disclosure: I'm the other of another paper on AD.)

I think JAX effectively demonstrates that this is indeed possible. The approach they use is to first linearise the JAXPR and then transpose it, pretty much in the same fashion as the Elliot paper did.

A JAXPR is a normal functional-style expression tree, with free variables, like

    let b = f a in g (a, b)
Elliot's "compiling to categories" requires you to translate this to

    g ∘ (id × f) ∘ dup
It's pretty baffling to work with terms like the latter in practice! The main ideas that JAX is based on were already published in "Lambda the ultimate backpropagator" in 2008. Elliot's work is a nice way of conceptualising the AD transformations and understanding how they all relate to each other, but it's not particularly practical.

Re: Differentiable Programming – A Simple Introduction

#43

Post by geysersam »

potbelly83 wrote: post #15
yauneyz wrote: post #3

My professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use that to optimize. Deep learning is just one application of that, but there are tons more.

How do you differentiate a string? Enum?

Not all functions are differentiable.

Sometimes there are other better ways to describe "how does changing x affect y". Derivatives are powerful but they are not the only possible description of such relationships.

I'm very excited for what other things future "compilers" will be able to do to programs besides differentiation. That's just the beginning.

Re: Differentiable Programming – A Simple Introduction

#44

Post by ChrisRackauckas »

choeger wrote: post #29

Nice article, but the intro is a little lengthy. I have one remark, though: If your language allows for automatic differentiation already, why do you bother with a neural network in the first place? I think you should have a good reason why you choose a neural network for your approximation of the inverse function and why it has exactly that amount of layers. For instance, why shouldn't a simple polynomial suffice? C…

It's a good question, which I answer in much more depth here: https://www.reddit.com/r/MachineLearning/comments/ryw53x/d_f... . Specifically that answer is about NNs vs fourier series, but it's the same point: polynomials, Fourier series, and NNs are all universal function approximators, so why use a NN? If you write out what happens though, you can easily see the curse of dimensionality in action and see why a 100-dimensional function shouldn't be approximated by tensor products of polynomials. See the rest there.

So it's not really about NNs vs polynomials/fourier/Chebyshev, etc., but rather what is the right thing to use at a given time. In the universal differential equations paper (https://arxiv.org/abs/2001.04385), we demonstrate that some examples of automatic equation discovery are faster using Fourier series than NNs (specifically there the discovery of a semilinear partial differential equation). That doesn't say that NNs are a bad way to do all of this, it just means that they are one trainable object that is good for high dimensional approximations, but libraries should allow you to easily move between classical basis functions and NNs to best achieve the highest performance.

Re: Differentiable Programming – A Simple Introduction

#45

Post by choeger »

ChrisRackauckas wrote: post #44
choeger wrote: post #29

Nice article, but the intro is a little lengthy. I have one remark, though: If your language allows for automatic differentiation already, why do you bother with a neural network in the first place? I think you should have a good reason why you choose a neural network for your approximation of the inverse function and why it has exactly that amount of layers. For instance, why shouldn't a simple polynomial suffice? C…

It's a good question, which I answer in much more depth here: https://www.reddit.com/r/MachineLearning/comments/ryw53x/d_f... . Specifically that answer is about NNs vs fourier series, but it's the same point: polynomials, Fourier series, and NNs are all universal function approximators, so why use a NN? If you write out what happens though, you can easily see the curse of dimensionality in action and see why a 100-d…

Great answer! Very much appreciated!

Re: Differentiable Programming – A Simple Introduction

#46

Post by celrod »

SleekEagle wrote: post #10
mountainriver wrote: post #8

Earlier quoted context omitted.

Julia wasn’t really built specifically to be differentiable, it was just built in a way that you have access to the IR, which is what zygote does. Enzyme AD is the most exciting to me because any LLVM language can be differentiable

Ah I see, thank you for clarifying. And thank you for bringing Enzyme to my attention - I've never seen it before!

Enzyme.jl works quite well (but the possibility of using it across languages is appealing).

Re: Differentiable Programming – A Simple Introduction

#47

Post by bmc7505 »

potbelly83 wrote: post #15
yauneyz wrote: post #3

My professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use that to optimize. Deep learning is just one application of that, but there are tons more.

How do you differentiate a string? Enum?

https://en.wikipedia.org/wiki/Brzozowski_derivative

Re: Differentiable Programming – A Simple Introduction

#48

Post by fennecs »

Does someone have an example where the ability to “differentiate” a program gets you something interesting?

I understand perfectly what it means for a neural network, but how about more abstract things.

Im not even sure as currently presented, the implementation actually means something. What is the derivative of a function like List, or Sort or GroupBy etc? These articles all assume that somehow it just looks like derivative from calculus somehow.

Approximating everything as some non smooth real function doesn’t seem entirely morally correct. A program is more discrete or synthetic. I think it should be a bit more algebraic flavoured, like differentials over a ring.

Re: Differentiable Programming – A Simple Introduction

#49

Post by dreamcompiler »

SleekEagle wrote: post #20
melony wrote: post #18

Earlier quoted context omitted.

Control theory has a very, very long parallel history alongside ML. ML, specifically probabilistic and reinforcement learning, uses a lot of dynamic programming ideas and Bellman equations in its theoretical modeling. Lookup the term cybernetics, it is an old term in the pre-internet era to mean control theory and optimization. The Soviets even had a grand scheme to build networked factories that could be centrally o…

This is some insanely cool history! I had no idea the Soviets had such a technical vision, that's actually pretty amazing. I've heard the term "cybernetics" but honestly just thought it was some movie-tech term, lol. It seems really weird that control theory is in EE departments considering it's sooo much more mathematical than most EE subdisciplines except signals processing. I remember a math professor of mine tell…

Laplace transforms are one such trick. Given a linear differential equation describing your system, Laplace transforms let you solve it using basic algebra. Unfortunately this doesn't work on nonlinear systems.

Re: Differentiable Programming – A Simple Introduction

#50

Post by melony »

dreamcompiler wrote: post #49
SleekEagle wrote: post #20

Earlier quoted context omitted.

This is some insanely cool history! I had no idea the Soviets had such a technical vision, that's actually pretty amazing. I've heard the term "cybernetics" but honestly just thought it was some movie-tech term, lol. It seems really weird that control theory is in EE departments considering it's sooo much more mathematical than most EE subdisciplines except signals processing. I remember a math professor of mine tell…

Laplace transforms are one such trick. Given a linear differential equation describing your system, Laplace transforms let you solve it using basic algebra. Unfortunately this doesn't work on nonlinear systems.

The reduction of convolution to FFT is beautiful ;)
Post reply on HN