How does this compare to zygote ?
Nilang.jl – A Reversible Julia DSL
11–20 of 31 posts
Re: Nilang.jl – A Reversible Julia DSL
#12,,The performance of reversible programming automatic differentiation is much better than most traditional frameworks.''
It would be good to see a ResNet training benchmark comparison with PyTorch as an example if this is really true.
Re: Nilang.jl – A Reversible Julia DSL
#13Incredible. Given a (one-to-one) Julia function `f`, this package generates `~f`, the inverse of f. Clever use of automatic differentiation. Paper: https://arxiv.org/abs/2003.04617
Forgive my ignorance, but what applications would this be used in where just maintaining copies of the initial parameters to f() wouldn’t work?
One can use this to express a large part of a deep ML model as a reversible function, thus drastically lowering memory requirements for backpropagation. The same technique, but applied by hand was recently used to speed up training for transformer language model: https://arxiv.org/abs/2001.04451
I think in the future we will see a trend of expressing most of DL model as reversible computation, with minimal irreversible module in the end and in the beginning.
Re: Nilang.jl – A Reversible Julia DSL
#14Incredible. Given a (one-to-one) Julia function `f`, this package generates `~f`, the inverse of f. Clever use of automatic differentiation. Paper: https://arxiv.org/abs/2003.04617
Is this in essence the same as the relation between a differential equation and it’s adjoint equation, and how one could use the asking method to perform back propagation? (Eg: ref. Neural ODEs)
NiLang requires every instruction being reversible, e.g. `SWAP` and `y += f(args...)`. The back-propagation is implemented on a limited instruction set. It is different with Neural ODE because neural ODE does not back propagate the program step by step. Some time reversible integrator can utilize the reversibility (with NiLang) to save memory. e.g. the leap frog integrator for symplectic systems.
Re: Nilang.jl – A Reversible Julia DSL
#15How does this compare to zygote ?
As far as I understand it (which is not very far):
Zygote calculates derivatives using source-to-source automatic differentiation.
This calculates function inverses (so to stretch the analogy a bit, it's kinda like "source-to-source automatic inversion")
Re: Nilang.jl – A Reversible Julia DSL
#16Wait a second, does it yield an exact inverse or a numerical approximation?
Very good question. Floating point `+=` and `-=` are not exactly reversible. This is the only approximation that we have made in NiLang. To compile NiLang to a reversible device (rigorously reversible), we need to overhaul current number systems, i.e. using fixed point numbers and logarithmic numbers instead. Fixed point numbers are exactly reversible under `+=` and `-=`, logarithmic numbers are exactly reversible under `*=` and `/=`. Here is an example of implementing Bessel function with two number systems: https://giggleliu.github.io/NiLang.jl/dev/examples/besselj/
Re: Nilang.jl – A Reversible Julia DSL
#17How does this compare to zygote ?
1. Scalar level
Various benchmarks (including those in the paper) show NiLang is much better than Zygote to differentiate scalar functions. And Zygote is much faster than TF and PyTorch.
2. Tensor level
Zygote, TF and PyTorch are much better than NiLang, because NiLang's matrix multiplication is not fully optimized, it is much slower than BLAS. (One can wrap BLAS into NiLang, but that does not measure NiLang's programming language level AD performance anymore)
Re: Nilang.jl – A Reversible Julia DSL
#18Incredible. Given a (one-to-one) Julia function `f`, this package generates `~f`, the inverse of f. Clever use of automatic differentiation. Paper: https://arxiv.org/abs/2003.04617
Thanks for the link to the paper, also check the latest version here: https://github.com/GiggleLiu/nilangpaper/blob/master/invc.pd...
Re: Nilang.jl – A Reversible Julia DSL
#19Earlier quoted context omitted.
Forgive my ignorance, but what applications would this be used in where just maintaining copies of the initial parameters to f() wouldn’t work?
I think the space requirements are different. Even if you only care about previous applications of f. Keeping snapshots of every input isn't always feasible.
Snapshots can be freed through reverse computing in NiLang. Reverse computing has subtle relationship with checkpointing, see author's blog: https://nextjournal.com/giggle/reverse-checkpointing .
Re: Nilang.jl – A Reversible Julia DSL
#20This is the latest NiLang tutorial notebook: https://github.com/JuliaReverse/NiLangTutorial