Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Enzyme – High-performance automatic differentiation of LLVM

enzyme.mit.edu

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Enzyme – High-performance automatic differentiation of LLVM

#31

Post by mhh__ »

wsmoses wrote: post #27
mhh__ wrote: post #22

Earlier quoted context omitted.

That last line doesn't make much sense to my eye at least.

I think in essence what PartiallyTyped is trying to say is that one potential optimization opportunity in whole-program AD is that you can avoid having to cache the original inputs of the program if you know that derivative computation won't need it (e.g. its only used in a sum and not a product or something whose derivative depends on the value). Some ML frameworks must cache all of the inputs to an operation since…

I think it was just a poorly stated description of the algebra of the derivative - I got the rest of it.

Re: Enzyme – High-performance automatic differentiation of LLVM

#32

Post by wsmoses »

ksr wrote: post #4

> The Enzyme project is a tool for performing reverse-mode automatic differentiation (AD) of statically-analyzable LLVM IR. This allows developers to use Enzyme to automatically create gradients of their source code without much additional work. Can someone please explain applications of creating gradients of my source code?

Say you have some existing virus simulation codebase that you want to use ML on to derive an effective policy on. Without an AD tool like Enzyme, you'd have to spend significant time and effort understanding and rewriting that obnoxious 100K lines of fortran into TensorFlow, when you could've been spending it solving your problem. The reason you need to do this rewriting is because many ML algorithms require the derivatives of functions to be able to use them and Enzyme provides an easy way to generate derivatives of existing code.

This is also useful in the scientific world where derivatives of functions are commonplace.

You could also use it in more performance-engineering/computer systems ways as well by using the derivatives to perform uncertainty quantification and perhaps decide to use 32-bit floats rather than 64-bit doubles.

Re: Enzyme – High-performance automatic differentiation of LLVM

#33

Post by wsmoses »

mhh__ wrote: post #31
wsmoses wrote: post #27

Earlier quoted context omitted.

I think in essence what PartiallyTyped is trying to say is that one potential optimization opportunity in whole-program AD is that you can avoid having to cache the original inputs of the program if you know that derivative computation won't need it (e.g. its only used in a sum and not a product or something whose derivative depends on the value). Some ML frameworks must cache all of the inputs to an operation since…

I think it was just a poorly stated description of the algebra of the derivative - I got the rest of it.

Yeah my best guess at that is that they were trying to say you'd only need to store one value: the sum, rather than the two individual values -- but I'm not completely sure.

Re: Enzyme – High-performance automatic differentiation of LLVM

#34

Post by wsmoses »

kwertzzz wrote: post #23
vchuravy wrote: post #13

One of the authors here, happy to answer any questions! (Particularly about the Julia integration)

The julia binding look really great. In zygote, arrays are immutable. Is enzyme able to handle such cases?

Enzyme does indeed handle mutable arrays (both in Enzyme.jl and any other frontend)! If you want to try it out forewarned that we're currently upgrading Enzyme.jl for better JIT integration (dynamic re-entry, custom derivative passthrough, nicer garbage collection) so there may be some falling debris.

Re: Enzyme – High-performance automatic differentiation of LLVM

#35

Post by PartiallyTyped »

mhh__ wrote: post #22
PartiallyTyped wrote: post #9

Earlier quoted context omitted.

It constructs an analytical gradient from the code. The reason is that you can compute the gradient directly. This can enable optimizations such as avoiding caching big matrices because you don't need to keep track of states/trace the graph, or you can compute the 2nd, 3rd, 4th... and so on derivatives because you have an analytical gradient. For example in an affine function, the gradient of the bias/intercept is th…

That last line doesn't make much sense to my eye at least.

The essence of what I was trying to say with the example is that a layer may be used multiple times through the computation graph. Without an analytical gradient, you may end up caching all of the inputs to the layer to compute the gradient. The alternative is to sum up the inputs because the gradient is linear with respect to the inputs; with an analytical gradient you can find that and compute it within the code instead of looking for adhoc optimisations within the graph.

Re: Enzyme – High-performance automatic differentiation of LLVM

#36

Post by PartiallyTyped »

wsmoses wrote: post #33
mhh__ wrote: post #31

Earlier quoted context omitted.

I think it was just a poorly stated description of the algebra of the derivative - I got the rest of it.

Yeah my best guess at that is that they were trying to say you'd only need to store one value: the sum, rather than the two individual values -- but I'm not completely sure.

That was my attempt.

Re: Enzyme – High-performance automatic differentiation of LLVM

#37

Post by shoyer »

wsmoses wrote: post #28
cl3misch wrote: post #24

Earlier quoted context omitted.

Isn't the input of the layer fundamentally a part of the gradient computation? So even in this case (inspecting LLVM code) the computation still needs to look at the input.

You don't always need the input to compute the gradient. For example the gradient of a sum function doesn't require the original input, it just sets all of the derivative(input)'s to 1.

To be more precise, in backwards mode auto-diff, inputs only need to be saved if they are used in a non-linear way.

Re: Enzyme – High-performance automatic differentiation of LLVM

#38

Post by PartiallyTyped »

cl3misch wrote: post #24
PartiallyTyped wrote: post #9

Earlier quoted context omitted.

It constructs an analytical gradient from the code. The reason is that you can compute the gradient directly. This can enable optimizations such as avoiding caching big matrices because you don't need to keep track of states/trace the graph, or you can compute the 2nd, 3rd, 4th... and so on derivatives because you have an analytical gradient. For example in an affine function, the gradient of the bias/intercept is th…

Isn't the input of the layer fundamentally a part of the gradient computation? So even in this case (inspecting LLVM code) the computation still needs to look at the input.

Even so, you maybe able to perform optimizations that were not possible under normal circumstances, e.g. you have an exponent in the output of a layer followed by a log in the next. Think SoftMax and logloss.

Re: Enzyme – High-performance automatic differentiation of LLVM

#39

Post by shoyer »

wsmoses wrote: post #32
ksr wrote: post #4

> The Enzyme project is a tool for performing reverse-mode automatic differentiation (AD) of statically-analyzable LLVM IR. This allows developers to use Enzyme to automatically create gradients of their source code without much additional work. Can someone please explain applications of creating gradients of my source code?

Say you have some existing virus simulation codebase that you want to use ML on to derive an effective policy on. Without an AD tool like Enzyme, you'd have to spend significant time and effort understanding and rewriting that obnoxious 100K lines of fortran into TensorFlow, when you could've been spending it solving your problem. The reason you need to do this rewriting is because many ML algorithms require the deri…

I'm a big believer in auto-diff, but I'm skeptical that any autodiff tool would differentiate a 100k line simulation code correctly and efficiently without manual intervention. I'd certainly love to be proven wrong, though and absolutely AD can be a big time saver :)

Re: Enzyme – High-performance automatic differentiation of LLVM

#40

Post by wsmoses »

shoyer wrote: post #39
wsmoses wrote: post #32

Earlier quoted context omitted.

Say you have some existing virus simulation codebase that you want to use ML on to derive an effective policy on. Without an AD tool like Enzyme, you'd have to spend significant time and effort understanding and rewriting that obnoxious 100K lines of fortran into TensorFlow, when you could've been spending it solving your problem. The reason you need to do this rewriting is because many ML algorithms require the deri…

I'm a big believer in auto-diff, but I'm skeptical that any autodiff tool would differentiate a 100k line simulation code correctly and efficiently without manual intervention. I'd certainly love to be proven wrong, though and absolutely AD can be a big time saver :)

Whoops added one too many zero’s there, agreed that would be really nice :P
Post reply on HN