From automatic differentiation to message passing [video]

Post by taliesinb » Thu, Aug 15, 2019, 11:16 PM UTC

taliesinb wrote: post #4

This is a perspective I’ve been taking recently as I’ve been investigating next-gen DL frameworks like Julia’s Zygote and Swift for Twnsorflow. So it’s very nice to see this articulated so well here by someone who has been thinking deeply about it for so long. I’ll add some things that relate to this perspective: there is a follow up paper to the DL via Hessian-free optimization paper by James Martens that develops a…

> For example, would it be useful to have an AD-like pass that calculated trusted regions for gradient updates?

To answer my own question, yes: http://papers.nips.cc/paper/7112-scalable-trust-region-metho...

Post by aqi » Fri, Aug 16, 2019, 6:43 AM UTC

At the end of the presentation the presented mentions that this is structurally identical to loopy belief propagation... Isn't that a big issue, since they inherit many of its tractability issues with regards to training and inference? Modern DL models are far too interconnected for inference to be tractable in general, so the best we can hope for is that we can make simplifying assumptions that make loopy belief propagation feasible.

As a side note, when modern compilers optimize abstract syntax trees, I'm pretty sure they do operations that are similar to the message-passing algorithm described. And they work great, albeit for specialized purposes.

Re: From automatic differentiation to message passing [video]

Re: From automatic differentiation to message passing [video]