The State of Probabilistic Programming

Post by mjw » Sun, Apr 12, 2015, 11:27 PM UTC

x0x0 wrote: post #21

cf wrote: post #11

What's interesting about most complaints of these systems is people talk about their poor performance or scalability? That is usually more a consequence of using MCMC or other inference algorithm than the language itself. MCMC is a very slow inference algorithm. Its primary advantage was that for well-known models it could be coded up much more simply than a fancier inference technique. When you consider variational…

Ease of coding is not the primary advantage of mcmc. The (giant) disadvantage of variational inference is the user needs to derive equations and understand quite a bit of math, where gibbs or other samplers like those in bugs/openbugs/jags/stan can work with the factored distributions and require much less mathematical sophistication from users.

Post by x0x0 » Sun, Apr 12, 2015, 11:40 PM UTC

mjw wrote: post #22

x0x0 wrote: post #21

Earlier quoted context omitted.

Ease of coding is not the primary advantage of mcmc. The (giant) disadvantage of variational inference is the user needs to derive equations and understand quite a bit of math, where gibbs or other samplers like those in bugs/openbugs/jags/stan can work with the factored distributions and require much less mathematical sophistication from users.

I know they've traditionally been quite fiddly, but I'm pretty sure computers can be persuaded to help derive the maths for variational methods these days. Perhaps a more important difference is that MCMC, while slow, is exact in the limit. Variational methods won't converge to the true posterior no matter how long you run them. You'll converge to an approximate answer which depends on the particular variational form…

Post by strmpnk » Mon, Apr 13, 2015, 1:21 AM UTC

vezzy-fnord wrote: post #8

enupten wrote: post #2

I wonder if kids 30y from now look upon this as the new Prolog.

I was pretty much thinking about logic programming throughout most of the article. Seems like PPS is a continuation of these ideas, in many ways.

Post by x0x0 » Mon, Apr 13, 2015, 2:36 AM UTC

davmre wrote: post #25

x0x0 wrote: post #24

Earlier quoted context omitted.

Are you aware of any work (or researchers) working on that? I would be very interested. And I think your second sentence reinforces my point -- it makes variational methods either more fiddly, or require more understanding to use well. edit: here's one such (limited but nice) effort: http://ebonilla.github.io/papers/nguyen-bonilla-nips-2014.pd...

Microsoft's Infer.NET is essentially an automated system for variational (/expectation-propagation) inference. It implements primitives for common operations and distributions, and then uses the local structure of mean-field inference ("variational message passing": http://www.jmlr.org/papers/volume6/winn05a/winn05a.pdf ) to build up variational inference algorithms for arbitrary factor graphs. It's not infinitely fl…

Post by stdbrouw » Mon, Apr 13, 2015, 9:08 AM UTC

x0x0 wrote: post #20

stdbrouw wrote: post #17

Earlier quoted context omitted.

Well, MCMC is pretty close to brute force so it's never going to be very fast. But the diagnostic tools are really quite good these days – tools like Stan and PyMC make it easy to see diagnostic plots that allow you to check for convergence and such. Additionally, more and more fine-tuning is happening automatically, like setting the right jump sizes and a plausible initial value for the random walk. Note that tradit…

I'm more pessimistic. My experience has been that inferring often benefits from reparameterization or analytic marginalizing, plus a bag of tricks that many people don't know. It's unfortunately still a long way from being a technology that you can pretty blindly use like ols or logistic regression. Particularly if you only care about prediction, not parameter inference, lr with a penalty search is pretty straightfor…