Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Foundations for Efficient and Expressive Differentiable Programming [pdf]

papers.nips.cc

Post reply on HN
Print view View original

1–10 of 15 posts

Re: Foundations for Efficient and Expressive Differentiable Programming [pdf]

#3

Post by tiarkrompf »

tiarkrompf wrote: post #2

Co-author here. Happy to answer any questions, as usual ...

We also have a more recent and slightly longer draft with additional explanations and GPU training results for ResNet 50 and DeepSpeech2:

"Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator" https://www.cs.purdue.edu/homes/rompf/papers/wang-preprint20...

The Lantern framework is available here: https://github.com/feiwang3311/Lantern

Re: Foundations for Efficient and Expressive Differentiable Programming [pdf]

#4

Post by improbable22 »

tiarkrompf wrote: post #2

Co-author here. Happy to answer any questions, as usual ...

This looks interesting, thanks.

What you call "delimited continuations" sounds a bit like how AD works in Julia's Flux package (and maybe elsewhere): During the forward calculation, a chain of functions is constructed, whose evaluation is the backward pass. This is done by overloading the original * to return both x * y and a closure Δ -> (Δ * y, x * Δ).

Does that sound right, are these indeed similar, or have I mis-understood something? I have never read Scala, but if I squint I can make your overloading of * is doing something similar.

Re: Foundations for Efficient and Expressive Differentiable Programming [pdf]

#5

Post by tiarkrompf »

improbable22 wrote: post #4
tiarkrompf wrote: post #2

Co-author here. Happy to answer any questions, as usual ...

This looks interesting, thanks. What you call "delimited continuations" sounds a bit like how AD works in Julia's Flux package (and maybe elsewhere): During the forward calculation, a chain of functions is constructed, whose evaluation is the backward pass. This is done by overloading the original * to return both x * y and a closure Δ -> (Δ * y, x * Δ). Does that sound right, are these indeed similar, or have I mis-…

It's different -- instead of returning a closure, we take a closure as additional parameter (check paper for details). This means that the call stack stays intact and all intermediate values can be stack-allocated.

Re: Foundations for Efficient and Expressive Differentiable Programming [pdf]

#6

Post by cs702 »

tiarkrompf wrote: post #2

Co-author here. Happy to answer any questions, as usual ...

Very cool.

Like all really good ideas, this one seems "obvious" in hindsight. I mean that is a compliment: It would have never occurred to me that transforming code into continuation-passing-style code would allow for automatic differentiation through all dynamic control-flow structures, by leveraging the function-call stack, thus eliminating the need for some kind of "tape" data structure, e.g., as in PyTorch.

My question is about the ongoing work to provide a JIT compiler for Python code. Do you expect it will provide full support for the entire PyTorch and/or Tensorflow APIs?

Re: Foundations for Efficient and Expressive Differentiable Programming [pdf]

#7

Post by improbable22 »

tiarkrompf wrote: post #5
improbable22 wrote: post #4

Earlier quoted context omitted.

This looks interesting, thanks. What you call "delimited continuations" sounds a bit like how AD works in Julia's Flux package (and maybe elsewhere): During the forward calculation, a chain of functions is constructed, whose evaluation is the backward pass. This is done by overloading the original * to return both x * y and a closure Δ -> (Δ * y, x * Δ). Does that sound right, are these indeed similar, or have I mis-…

It's different -- instead of returning a closure, we take a closure as additional parameter (check paper for details). This means that the call stack stays intact and all intermediate values can be stack-allocated.

OK, thanks, sounds worth trying to wrap my head around...

(Note BTW that the github page has a dead link to the paper.)

Re: Foundations for Efficient and Expressive Differentiable Programming [pdf]

#8

Post by tiarkrompf »

cs702 wrote: post #6
tiarkrompf wrote: post #2

Co-author here. Happy to answer any questions, as usual ...

Very cool. Like all really good ideas, this one seems "obvious" in hindsight. I mean that is a compliment: It would have never occurred to me that transforming code into continuation-passing-style code would allow for automatic differentiation through all dynamic control-flow structures, by leveraging the function-call stack, thus eliminating the need for some kind of "tape" data structure, e.g., as in PyTorch. My qu…

Thanks! Yes, it wasn't obvious at all when we started looking at AD either.

Lantern supports a good deal of PyTorch (via Snek, our Python front-end similar to AutoGraph) and can also read ONNX. Full feature parity is not our main goal--so far, supported features have been driven mostly by what is required for certain interesting models.

Re: Foundations for Efficient and Expressive Differentiable Programming [pdf]

#9

Post by tiarkrompf »

improbable22 wrote: post #7
tiarkrompf wrote: post #5

Earlier quoted context omitted.

It's different -- instead of returning a closure, we take a closure as additional parameter (check paper for details). This means that the call stack stays intact and all intermediate values can be stack-allocated.

OK, thanks, sounds worth trying to wrap my head around... (Note BTW that the github page has a dead link to the paper.)

Thanks! Fixed now.
Post reply on HN