Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Differentiable Programming System to Bridge ML and Scientific Computing

arxiv.org

Post reply on HN
Print view View original

61–70 of 75 posts

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#61

Post by skybrian »

KenoFischer wrote: post #50
skybrian wrote: post #33

This looks great! The language features of Julia seems to be exploited to good effect. However, I'm curious: are there any aspects of Julia's design that, in retrospect, make differential programming harder or more inconvenient?

There's two competing currents here. One the on hand, Julia is extremely powerful and dynamic, so it has very high expressability for any possible differentiable programming you could think of. It also has a fairly simple core, so as long as you know how to properly transform the core, you can get a mathematically correct differential. However, the reason julia works so well is that the compiler is able to understand…

Thanks! So it sounds like what you're saying is that, as with other languages that have complex optimizers, there can be performance cliffs when you get too far off the beaten path?

For someone not working on the Julia compiler, how tricky it is to figure out what to do to improve performance?

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#62

Post by steve76 »

_ks3e wrote: post #59
KenoFischer wrote: post #45

Earlier quoted context omitted.

Zygote is an orthogonal piece of technology on this front and relies on a good optimizing compiler behind it to target actual hardware. Its focus is primarily on expressability. We've been talking about automatic kernel generation for a while (and when I saw kernel generation what I mean is basically search for access patterns), but note that it's not quite as bad a problem in julia, because you can use higher order…

In terms of trying to break free of dependence on hand optimized kernels: a few people, myself included, have been working on some theoretical approaches to generating cache-efficient rearrangements for neutral net like problems. We've worked it out for convolution like problems [1] and have some upcoming results generalizing these techniques to other problems. Please feel free to email if you'd like to talk. [1] htt…

God damn you RMS.

Just do this please:

https://twitter.com/thisbounty_com/status/115239073901339443...

https://twitter.com/thisbounty_com/status/115239756590162329...

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#63

Post by eggy »

I keep orbiting Julia, because of developments like this one. I am an old Lisper/Scheme guy. How does this compare to say R6RS-AD[1] or cl-autodiff[2] or Deriva for Clojure/Java[3]?

I realize Julia has femtolisp inside, yet another Lisp bait for me!

[1] https://github.com/qobi/R6RS-AD

[2] https://github.com/masonium/cl-autodiff

[3] https://github.com/lambder/Deriva

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#64

Post by cs702 »

KenoFischer wrote: post #45
cs702 wrote: post #37

Earlier quoted context omitted.

Keno: first of all, let me give a big public thank you to you and your colleagues. (For those here who don't know, Keno is listed as one of the authors of the paper, and works closely with Mike Innes, lead author and also lead developer of Zygote. Mike is also an active member of HN.) Second, let me bring up what I think is a significant issue. My perception is that most deep learning researchers and practitioners --…

Zygote is an orthogonal piece of technology on this front and relies on a good optimizing compiler behind it to target actual hardware. Its focus is primarily on expressability. We've been talking about automatic kernel generation for a while (and when I saw kernel generation what I mean is basically search for access patterns), but note that it's not quite as bad a problem in julia, because you can use higher order…

Thank you for your response. Makes sense, and I'm happy to hear you and others are aware of the issue.

PS. I now feel that I asked my question without first thinking about it a bit more; sorry about that. I temporarily "forgot" that Zygote is a source-to-source AD package because, as someone who is developing and iterating over deep learning models for potential deployment to production, I naturally tend to think in terms of monolithic software stacks -- e.g.,"the TensorFlow stack," "the PyTorch stack," "the nascent Julia stack," and so on.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#65

Post by one-more-minute »

KenoFischer wrote: post #10
GistNoesis wrote: post #7

An alternative approach https://www.microsoft.com/en-us/research/video/the-simple-es... where you define the right functional structures that allow a very compact and efficient way of expressing automatic differentiation and then let the existing compiler do the heavy lifting.

I've read the corresponding paper, and while I think it's a fun read, I'm not sure it does particularly much to clear up the underlying confusion here. Reverse mode AD on straight line primitives is describabable in about three lines of exposition. The tricky part is how do you handle control flow (equivalently recursion). The description thereof then depends on what your underlying IR data structure is, but describi…

> As an aside, a pet peeve of mine is people calling these algorithms "tape free".

The common autograd-style tape combines a recording of both the operations of a program and its intermediate values, making the term ambiguous; so when people say "tape-free" they mean avoiding recorded operations.

In the Julia world we've tried to disambiguate with "trace" and "tape" for operations and values, but that's not standard terminology, unfortunately.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#66

Post by one-more-minute »

KenoFischer wrote: post #51
roseandking wrote: post #38

How is this different from JAX?

JAX is a sophisticated and well implemented high level tracer, combined with a backend compiler that makes use of XLA. High level tracing has some fundamental draw backs in key features we want (efficient scalar AD for example), so the approach being advocated here is doing AD as a compiler transform on the original source language. Mike has some details in an earlier paper: https://arxiv.org/abs/1810.07951 . As a sh…

The really big problem with tracing, `@jit` annotations, and similar approaches, is that you're no longer running Python, but something almost-but-not-quite Python – modifying the language's semantics is necessary to get performance.

In deep learning, tweaking some syntax and adding some annotations isn't a big deal, but for the use cases we're interested in we really don't want to rewrite every library to be AD compatible. Zygote is pretty unique (outside of the scientific computing world) in being able to take libraries that were written years before AD existed in Julia, and differentiate them correctly and efficiently.

There are other, more subtle and technical, issues with those approaches, but that's really the Big Deal.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#67

Post by ddragon »

eggy wrote: post #63

I keep orbiting Julia, because of developments like this one. I am an old Lisper/Scheme guy. How does this compare to say R6RS-AD[1] or cl-autodiff[2] or Deriva for Clojure/Java[3]? I realize Julia has femtolisp inside, yet another Lisp bait for me! [1] https://github.com/qobi/R6RS-AD [2] https://github.com/masonium/cl-autodiff [3] https://github.com/lambder/Deriva

Those are based on overloading functions to add the new behaviors (computing the forward diff or building the graph within a special variable for the reverse mode) which in Julia is implemented for example in [1], [2], [3] and [4], while the one in OP is based on source analysis and transformation.

And the similarities with Lisp are more than just the parser being written in it, Julia's programming paradigm is based on the CLOS, everything is an expression, hygienic macros and reader macros and code is data. It mostly just lacks sexpr.

[1] https://github.com/JuliaDiff/ForwardDiff.jl

[2] https://github.com/dfdx/Yota.jl

[3] https://github.com/denizyuret/AutoGrad.jl (from the Knet framework)

[4] https://fluxml.ai/Flux.jl/stable/internals/tracker/ (from the Flux framework)

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#68

Post by tel »

shoyer wrote: post #34
tel wrote: post #20

Earlier quoted context omitted.

Bridging into NumPy isn’t going to be useful unless it covers some AD system too (XLA?).

XLA is a compiler for array code. It doesn't come with AD -- you need a wrapper like TF or JAX (or Swift, I guess) for that.

I was imagining the FFI emitting XLA code needed to implement the TF/JAX routines behind the scenes, but it's wild speculation.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#69

Post by KenoFischer »

one-more-minute wrote: post #65
KenoFischer wrote: post #10

Earlier quoted context omitted.

I've read the corresponding paper, and while I think it's a fun read, I'm not sure it does particularly much to clear up the underlying confusion here. Reverse mode AD on straight line primitives is describabable in about three lines of exposition. The tricky part is how do you handle control flow (equivalently recursion). The description thereof then depends on what your underlying IR data structure is, but describi…

> As an aside, a pet peeve of mine is people calling these algorithms "tape free". The common autograd-style tape combines a recording of both the operations of a program and its intermediate values, making the term ambiguous; so when people say "tape-free" they mean avoiding recorded operations. In the Julia world we've tried to disambiguate with "trace" and "tape" for operations and values, but that's not standard…

Yes, I understand that, but i still think it's misleading. Even in Julia, we're still recording operations, just a) at a much coarser granularity (if there's no control flow in the middle we can record it as one operation) b) in the type of the tape data structure rather than explicitly

For example, one could imagine applying common subexpression compression to a tracer tape, and would get essentially the same thing.

Other places to stores this information are the stack of a closure chain, but it's still fundamentally the same information.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#70

Post by one-more-minute »

KenoFischer wrote: post #69
one-more-minute wrote: post #65

Earlier quoted context omitted.

> As an aside, a pet peeve of mine is people calling these algorithms "tape free". The common autograd-style tape combines a recording of both the operations of a program and its intermediate values, making the term ambiguous; so when people say "tape-free" they mean avoiding recorded operations. In the Julia world we've tried to disambiguate with "trace" and "tape" for operations and values, but that's not standard…

Yes, I understand that, but i still think it's misleading. Even in Julia, we're still recording operations, just a) at a much coarser granularity (if there's no control flow in the middle we can record it as one operation) b) in the type of the tape data structure rather than explicitly For example, one could imagine applying common subexpression compression to a tracer tape, and would get essentially the same thing.…

In a more traditional SCT AD like Tapenade, there isn't really anything that corresponds to recording of program operations at runtime. Zygote arguably does something semantically equivalent to recording a trace (at the interprocedural level), but of course the idea is that once you've optimised that away we approximate Tapenade – so the line is a bit blurred.

Don't get me wrong, I agree with your core point: "tape-free" conflates multiple things, and none of them really capture the AD design space in a useful way. Hopefully as the field settles down we'll figure out more useful axes for comparing these tools.

Post reply on HN