Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Differentiable Programming System to Bridge ML and Scientific Computing

arxiv.org

Post reply on HN
Print view View original

41–50 of 75 posts

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#41

Post by eigenspace »

kensai wrote: post #30

Cool and amazing. This thing should be used to enrich base Julia, not simply be an extension.

Julia is designed in such a way that third party 'extensions' (packages) are first class citizens. Very little actually needs to be in Base or the standard library. When one types `using Zygote` you're not using a programming language with a siloed AD exension. You've radically extenended julia into a differentiable programming language.

The beauty of Zygote.jl being a package is that we don't force one AD approach on the entire language ecosystem. Julia has several very promising AD systems in development, each of which is preferable for certain situations but are close enough in API that they're basically hot swappable.

When one of them needs a change made to base julia to better support the sort of compiler transformations they want to do, they make a PR and the new functionality gets implemented very quickly and all the other AD / compiler transformation packages benefit from the change.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#42

Post by phrmoy »

zitterbewegung wrote: post #23

This looks neat . In python is this also a project that does the same thing ? https://github.com/HIPS/autograd

Would also be curious to understand how this compares. They seem to account for recursion and if-branches as well.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#43

Post by kensai »

eigenspace wrote: post #41
kensai wrote: post #30

Cool and amazing. This thing should be used to enrich base Julia, not simply be an extension.

Julia is designed in such a way that third party 'extensions' (packages) are first class citizens. Very little actually needs to be in Base or the standard library. When one types `using Zygote` you're not using a programming language with a siloed AD exension. You've radically extenended julia into a differentiable programming language. The beauty of Zygote.jl being a package is that we don't force one AD approach o…

That's awesome, good to know! One more reason to learn Julia!

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#44

Post by eigenspace »

phrmoy wrote: post #42
zitterbewegung wrote: post #23

This looks neat . In python is this also a project that does the same thing ? https://github.com/HIPS/autograd

Would also be curious to understand how this compares. They seem to account for recursion and if-branches as well.

They're not really all that close. For instance that Autograd library requires that you use its own version of numpy instead of the regular version because it can't differentiate C. Because most non-trivial python code is actually written in C, there's almost no performant programs you can just differentiate out of the box unless autograd itself has a fork of that package.

My understanding is that the python Autograd library is also quite slow (though it's been a while since I looked at it).

Zygote will give the same runtime performance as handwritten derivatives in many cases and works across almost the entire julia ecosystem since nearly all julia packages are truly written in julia.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#45

Post by KenoFischer »

cs702 wrote: post #37
KenoFischer wrote: post #4

Happy that this paper finally made it to arxiv. The biggest reason for writing it was to try and showcase some of the breadth of applications we see for really high quality first class AD support at the language level. There are several communities that need this technology, so it makes sense to try and build one system that can address all of them and share tricks. I'm also hoping this gives people a sense of why ou…

Keno: first of all, let me give a big public thank you to you and your colleagues. (For those here who don't know, Keno is listed as one of the authors of the paper, and works closely with Mike Innes, lead author and also lead developer of Zygote. Mike is also an active member of HN.) Second, let me bring up what I think is a significant issue. My perception is that most deep learning researchers and practitioners --…

Zygote is an orthogonal piece of technology on this front and relies on a good optimizing compiler behind it to target actual hardware. Its focus is primarily on expressability. We've been talking about automatic kernel generation for a while (and when I saw kernel generation what I mean is basically search for access patterns), but note that it's not quite as bad a problem in julia, because you can use higher order functions to get a lot of composability (e.g. if there's a hand-optimized parameterized matmul, fusing in inputs and outputs is just passing in extra functions). There's some academic work at the JuliaLab that's promising. We're also in discussions with commercial partners who care about this kind of thing to see if somebody is willing to fund it, but so far existing solutions have been good enough while we work on higher priority items. I do agree that being limited to a few hand-tuned kernels is a significant drag on research, so I hope we can do something here.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#46

Post by KenoFischer »

cs702 wrote: post #40
ChrisRackauckas wrote: post #39

Earlier quoted context omitted.

There are people in the Julia Lab working on high level tensor operation languages and compilers. It's a hard problem but one that many are interested in solving with Julia.

Chris: As a user of these tools, I cannot tell you how thankful I am for the work you, Keno, Mike and others do. (For those who don't know, Chris works closely with Mike, Keno, and others in the Julia team.) I recognize that this is a hard problem.[a] FWIW, I read or heard (can't remember which) that there are people working with Chris Lattner seeking to use predictive AI (instead of search and heuristics) to address…

I'm not familiar with that part of the MLIR work, though I wouldn't be surprised if they are working at it. Google Brain is doing some of the finest work on ML for systems programming, so this'd be right up their alley. If it works well and they come up with some useful models, we'll be more than happy to incorporate them (or just use MLIR directly - there's a talk at JuliaCon next week from a Google engineer on using Julia as an MLIR frontend: https://pretalx.com/juliacon2019/talk/3YBZLC/).

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#47

Post by ddragon »

zitterbewegung wrote: post #23

This looks neat . In python is this also a project that does the same thing ? https://github.com/HIPS/autograd

Autograd (and most of the current approaches) work by having a special object for the data and overloaded methods that instead of immediately executing an operation they instead store it in a graph of transformations. Then when you need the gradient it applies the chain rule over this graph. The support for loops/control flow is possible since at each call you destroy and recreate the graph, which is not optimal for performance but makes it very dynamic (tensorflow eager/pytorch vs tensorflow graph interface).

That's also an approach that Julia excels because of multiple dispatch which you can see explained in [1].

In that case you have effectively two separate languages, the language used to generate the graph and the graph, each. This approach applies the transformation directly on the Julia IR to generate the gradient descent as if you wrote it directly on Julia side by side with the code that was written in a way that is completely unaware of that transformation (such as the ability to differentiate libraries that were built before that approach even existed). So the end product is something that is similar to the tensorflow graph (it has all control flow already embedded and can be pre-optimized by a compiler), but that is even easier to write than tensorflow eager (which is also the intent of Swift for Tensorflow).

[1] https://github.com/MikeInnes/diff-zoo

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#48

Post by KenoFischer »

jknz wrote: post #22

This is huge and I have high hopes for zygote,jl. Observation/request: For higher-order derivatives (Hessian, Laplacian, etc), AD libraries typically provide API shortcuts. I have found it difficult to control or predict the memory footprint and the time complexity of these API shortcuts. Laplacian is case in point: it is sometimes computed by first computing the Hessian by forward-over-reverse and then taking the tr…

Unfortunately, this isn't super easy since the time complexity heavily depends on what optimizations the compiler will apply and how the higher order AD is exactly implemented (there's many ways to do so that all give you the same answer and you probably want the system to pick the best for you). What we could do fairly easily however is to have a compiler introspection tool that runs through the computation, but doesn't do any of the work, just record how much work it would have done. That would easily give you and idea and let you plot a parameter sweep over any dimension you care about with actual results corresponding to the capabilities of the system.

Re: A Differentiable Programming System to Bridge ML and Scientific Computing

#50

Post by KenoFischer »

skybrian wrote: post #33

This looks great! The language features of Julia seems to be exploited to good effect. However, I'm curious: are there any aspects of Julia's design that, in retrospect, make differential programming harder or more inconvenient?

There's two competing currents here. One the on hand, Julia is extremely powerful and dynamic, so it has very high expressability for any possible differentiable programming you could think of. It also has a fairly simple core, so as long as you know how to properly transform the core, you can get a mathematically correct differential. However, the reason julia works so well is that the compiler is able to understand and eliminate all the dynamism and complexity for you at run time and generate very tight code as a result. When you add AD into the mix, the generated code becomes, much, much more complicated, so the compiler needs to work a lot harder. That can sometimes be a cause for frustration for us, because we get something really cool working with absolutely zero effort, but then need to go back and make the compiler better to reclaim some performance (static languages have the opposite problem, where you need to improve the type system in order to allow the thing to happen, but once you have described it in a specific enough type system, you usually also have good performance - if your language is any good that is).

Overall I think Julia is a pretty great language to implement AD (as evidenced I'd say by the 15 or so different AD packages that people have written for individual use cases before the latest push for a unified one), but it still is a very powerful language, so if you want your AD to handle the whole language (as we do), then you're gonna have to do a bit of work.

Post reply on HN