Viewing profile — wsmoses

The name of the LLVM AD tool is actually Enzyme [ http://enzyme.mit.edu/ ] (Zygote is a Julia tool)

Oh for sure, any ML framework worth its salt should do some amount of graph rewriting / transformations. I was (perhaps poorly) trying to explain how while yes AD (regardless of im…

Regarding differentiating python via CPython, theoretically yes, though practically it is likely more wise to use something like Numba which takes Python to LLVM directly to avoid …

We go into more details in the Limitations section of the paper, but in short Enzyme requires the following properties: * IR of active functions must be accessible when Enzyme is c…

Whoops added one too many zero’s there, agreed that would be really nice :P

Enzyme does indeed handle mutable arrays (both in Enzyme.jl and any other frontend)! If you want to try it out forewarned that we're currently upgrading Enzyme.jl for better JIT in…

Yeah my best guess at that is that they were trying to say you'd only need to store one value: the sum, rather than the two individual values -- but I'm not completely sure.

Say you have some existing virus simulation codebase that you want to use ML on to derive an effective policy on. Without an AD tool like Enzyme, you'd have to spend significant ti…

Reverse mode AD can always get into situations where it needs to store original values (i.e. network state). One advantage, however, of doing a more whole-program approach to AD ra…

Enzyme needs to be able to access the IR of any potentially active functions (calls that it deduced could impact the gradient) to be able to differentiate them. If all of the code …

You don't always need the input to compute the gradient. For example the gradient of a sum function doesn't require the original input, it just sets all of the derivative(input)'s …

I think in essence what PartiallyTyped is trying to say is that one potential optimization opportunity in whole-program AD is that you can avoid having to cache the original inputs…

Oh man that was a fun hack to write. Basically we demonstrated an easy-to-setup AD on rust by leveraging link-time optimization (LTO) as a way to make sure Enzyme's generate deriva…

For GPU's, there's a couple of different things that you might want to do. You can use existing tools within LLVM to automatically generate GPU code out of existing code, and this …

Adding onto this, numerical derivatives have two potential problems which is why they tend not to be used in big scientific/ML frameworks. First of all they suffer from accuracy de…

Enzyme is named such as it's a tool that "synthesizes derivatives" and also as a pun referencing Zygote (another AD tool) since Enzyme operates at a lower level (LLVM rather than J…

Hi all, another author here and happy to answer any questions! Some more relevant links for the curious Github: https://github.com/wsmoses/Enzyme Paper: https://proceedings.neurips…

Hi all, author here. A couple of relevant links for the curious Github: https://github.com/wsmoses/Enzyme Paper: https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b... Pro…