Differentiable Programming – A Simple Introduction
assemblyai.com
Differentiable Programming – A Simple Introduction
1–10 of 50 posts
Re: Differentiable Programming – A Simple Introduction
#2The nice thing about differentiable programming is that we can use all sorts of different optimizers compared to gradient descent that can offer quadratic convergence instead of linear!
Re: Differentiable Programming – A Simple Introduction
#3My professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use that to optimize. Deep learning is just one application of that, but there are tons more.
Re: Differentiable Programming – A Simple Introduction
#4The nice thing about differentiable programming is that we can use all sorts of different optimizers compared to gradient descent that can offer quadratic convergence instead of linear!
Yes exactly! This is huge. Hessian optimization is really easy with JAX, haven't tried it in Julia though
Re: Differentiable Programming – A Simple Introduction
#5My professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use that to optimize. Deep learning is just one application of that, but there are tons more.
That's part of why Julia is so exciting! Building it specifically to be a differentiable programming language opens so many doors ...
Re: Differentiable Programming – A Simple Introduction
#6My professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use that to optimize. Deep learning is just one application of that, but there are tons more.
I am just happy that the previously siloed fields of operations research and various control theory sub-disciplines are now incentivized to pool their research together thanks to the funding in ML. Also many expensive and proprietary optimization software in industry are finally getting some competition.
Re: Differentiable Programming – A Simple Introduction
#7My professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use that to optimize. Deep learning is just one application of that, but there are tons more.
I am just happy that the previously siloed fields of operations research and various control theory sub-disciplines are now incentivized to pool their research together thanks to the funding in ML. Also many expensive and proprietary optimization software in industry are finally getting some competition.
Hm I didn't know different areas of control theory were siloed. Learning about control theory in graduate school was awesome and it seems like a field that would benefit from ML a lot. I know they use RL agents for control networks for e.g. cartpole, but I would've thought it would be more widespread! Do you think the development of Differentiable Programming (i.e. the observation of more generality beyond pure ML/DL) was really the missing piece?
Also, just curious, what are your studies in?
Re: Differentiable Programming – A Simple Introduction
#8My professor has talked about this. He thinks that the real gem of the deep learning revolution is the ability to take the derivative of arbitrary code and use that to optimize. Deep learning is just one application of that, but there are tons more.
That's part of why Julia is so exciting! Building it specifically to be a differentiable programming language opens so many doors ...
Julia wasn’t really built specifically to be differentiable, it was just built in a way that you have access to the IR, which is what zygote does. Enzyme AD is the most exciting to me because any LLVM language can be differentiable
Re: Differentiable Programming – A Simple Introduction
#9The nice thing about differentiable programming is that we can use all sorts of different optimizers compared to gradient descent that can offer quadratic convergence instead of linear!
Yes exactly! This is huge. Hessian optimization is really easy with JAX, haven't tried it in Julia though
And very fast given that you compile the procedure! I am considering writing an article on this and posting it here because I have seen enormous improvements over non jitted code, and that excluded jax.vmap.
Re: Differentiable Programming – A Simple Introduction
#10Earlier quoted context omitted.
That's part of why Julia is so exciting! Building it specifically to be a differentiable programming language opens so many doors ...
Julia wasn’t really built specifically to be differentiable, it was just built in a way that you have access to the IR, which is what zygote does. Enzyme AD is the most exciting to me because any LLVM language can be differentiable
Ah I see, thank you for clarifying. And thank you for bringing Enzyme to my attention - I've never seen it before!