Earlier quoted context omitted.
Well, quadratic convergence usually requires the Hessian, or an approximation of it, and that's difficult to get in deep learning due to memory constrains, and difficulty computing second order derivatives. Computing the derivatives is not very difficult with e.g. Jax, but ... you get back to the memory issue. The Hessian is a square matrix, so in Deep Learning, if we have a million of parameters, then the Hessian is…
Not only does it have 1 trillion elements, you also have to invert it!
Differentiable Programming – A Simple Introduction
21–30 of 50 posts
Re: Differentiable Programming – A Simple Introduction
#22Earlier quoted context omitted.
Control theory has a very, very long parallel history alongside ML. ML, specifically probabilistic and reinforcement learning, uses a lot of dynamic programming ideas and Bellman equations in its theoretical modeling. Lookup the term cybernetics, it is an old term in the pre-internet era to mean control theory and optimization. The Soviets even had a grand scheme to build networked factories that could be centrally o…
This is some insanely cool history! I had no idea the Soviets had such a technical vision, that's actually pretty amazing. I've heard the term "cybernetics" but honestly just thought it was some movie-tech term, lol. It seems really weird that control theory is in EE departments considering it's sooo much more mathematical than most EE subdisciplines except signals processing. I remember a math professor of mine tell…
There is this excellent HN-recommended fiction called Red Plenty that dramatised the efforts on the other side of the Atlantic.
https://news.ycombinator.com/item?id=8417882
> It seems really weird that control theory is in EE departments considering it's sooo much more mathematical than most EE subdisciplines except signals processing.
I agree, apparently Bellman's reasoning for calling dynamic programming what it is was because he needed grant funding during the Cold War days and was advised to give his mathematical theories a more "interesting" name.
https://en.m.wikipedia.org/wiki/Dynamic_programming#History
The generalised form of the Bellman Equation (co-formulated by Kalman of the Kalman filters fame) to control theory and EE is in some ways what the Maximum Likelihood function is to ML.
https://en.m.wikipedia.org/wiki/Hamilton%E2%80%93Jacobi%E2%8...
Re: Differentiable Programming – A Simple Introduction
#23The most interesting thing I've seen on AD is "The simple essence of automatic differentiation" (2018) [1]. See past discussion [2], and talk [3]. I think the main idea is that by compiling to categories and pairing up a function with its derivative, the pair becomes trivially composable in forward mode, and the whole structure is easily converted to reverse mode afterwards. [1]: https://dl.acm.org/doi/10.1145/323676…
> the whole structure is easily converted to reverse mode afterwards.
Unfortunately it's not. Elliot never actually demonstrates in the paper how to implement such an algorithm, and it's very hard to write compiler transformations in "categorical form".
(Disclosure: I'm the other of another paper on AD.)
Re: Differentiable Programming – A Simple Introduction
#24Re: Differentiable Programming – A Simple Introduction
#25Earlier quoted context omitted.
This is some insanely cool history! I had no idea the Soviets had such a technical vision, that's actually pretty amazing. I've heard the term "cybernetics" but honestly just thought it was some movie-tech term, lol. It seems really weird that control theory is in EE departments considering it's sooo much more mathematical than most EE subdisciplines except signals processing. I remember a math professor of mine tell…
There is this excellent HN-recommended fiction called Red Plenty that dramatised the efforts on the other side of the Atlantic. https://news.ycombinator.com/item?id=8417882 > It seems really weird that control theory is in EE departments considering it's sooo much more mathematical than most EE subdisciplines except signals processing. I agree, apparently Bellman's reasoning for calling dynamic programming what it is…
Looks really cool, added to my amazon cart. Thanks for the rec!
That hilarious and sadly insightful. I remember thinking "what the hell is so 'dynamic' about this?" the first time I learned about dynamic programming. Although "memoitative programming" sounds pretty fancy too, lol
Re: Differentiable Programming – A Simple Introduction
#26The most interesting thing I've seen on AD is "The simple essence of automatic differentiation" (2018) [1]. See past discussion [2], and talk [3]. I think the main idea is that by compiling to categories and pairing up a function with its derivative, the pair becomes trivially composable in forward mode, and the whole structure is easily converted to reverse mode afterwards. [1]: https://dl.acm.org/doi/10.1145/323676…
> the whole structure is easily converted to reverse mode afterwards. Unfortunately it's not. Elliot never actually demonstrates in the paper how to implement such an algorithm, and it's very hard to write compiler transformations in "categorical form". (Disclosure: I'm the other of another paper on AD.)
I think JAX effectively demonstrates that this is indeed possible. The approach they use is to first linearise the JAXPR and then transpose it, pretty much in the same fashion as the Elliot paper did.
Re: Differentiable Programming – A Simple Introduction
#27The most interesting thing I've seen on AD is "The simple essence of automatic differentiation" (2018) [1]. See past discussion [2], and talk [3]. I think the main idea is that by compiling to categories and pairing up a function with its derivative, the pair becomes trivially composable in forward mode, and the whole structure is easily converted to reverse mode afterwards. [1]: https://dl.acm.org/doi/10.1145/323676…
> the whole structure is easily converted to reverse mode afterwards. Unfortunately it's not. Elliot never actually demonstrates in the paper how to implement such an algorithm, and it's very hard to write compiler transformations in "categorical form". (Disclosure: I'm the other of another paper on AD.)
which paper?
Re: Differentiable Programming – A Simple Introduction
#28Earlier quoted context omitted.
Well, quadratic convergence usually requires the Hessian, or an approximation of it, and that's difficult to get in deep learning due to memory constrains, and difficulty computing second order derivatives. Computing the derivatives is not very difficult with e.g. Jax, but ... you get back to the memory issue. The Hessian is a square matrix, so in Deep Learning, if we have a million of parameters, then the Hessian is…
Not only does it have 1 trillion elements, you also have to invert it!
Indeed! BFGS (and derivatives) approximate the inverse but they have other issues that make them prohibitively expensive.
Re: Differentiable Programming – A Simple Introduction
#29Nice article, but the intro is a little lengthy.
I have one remark, though: If your language allows for automatic differentiation already, why do you bother with a neural network in the first place?
I think you should have a good reason why you choose a neural network for your approximation of the inverse function and why it has exactly that amount of layers. For instance, why shouldn't a simple polynomial suffice? Could it be that your neural network ends up as an approximation of the Taylor expansion of your inverse function?
Re: Differentiable Programming – A Simple Introduction
#30Nice article, but the intro is a little lengthy. I have one remark, though: If your language allows for automatic differentiation already, why do you bother with a neural network in the first place? I think you should have a good reason why you choose a neural network for your approximation of the inverse function and why it has exactly that amount of layers. For instance, why shouldn't a simple polynomial suffice? C…
I think for more complicated examples like RL control systems a neural network is the natural choice. If you can incorporate physics into your world model then you'd need differentiable programming + NNs, right? Or am I misunderstanding the question.
If you're talking about the specific cannon problem, you don't need to do any learning at all you can just solve the kinematics, so in some sense you could ask why you're using any approximation function,