Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of Probabilistic Programming

moalquraishi.wordpress.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: The State of Probabilistic Programming

#11

Post by cf »

What's interesting about most complaints of these systems is people talk about their poor performance or scalability? That is usually more a consequence of using MCMC or other inference algorithm than the language itself.

MCMC is a very slow inference algorithm. Its primary advantage was that for well-known models it could be coded up much more simply than a fancier inference technique. When you consider variational methods and newer streaming methods based on things like Assumed Density Filtering you can get really great scalable performance. The point of probabilistic programming is write inference algorithms once for a large class of models and be done. So the advantage of using a fancier method is amplified.

This means paradoxically probabilistic programming should eventually be faster than existing methods rather than slower, since you can reuse these fancier inference methods for new models. This is a very active field so this progress is only starting to be appear in the existing systems.

Re: The State of Probabilistic Programming

#12

Post by detaro »

cf wrote: post #11

What's interesting about most complaints of these systems is people talk about their poor performance or scalability? That is usually more a consequence of using MCMC or other inference algorithm than the language itself. MCMC is a very slow inference algorithm. Its primary advantage was that for well-known models it could be coded up much more simply than a fancier inference technique. When you consider variational…

> The point of probabilistic programming is write inference algorithms once for a large class of models and be done. So the advantage of using a fancier method is amplified.

So that should be something where a few standard libraries or toolsets should emerge that push out the "easy-to-implement" default choice? Are there any contenders yet? (As you might be able to tell, I don't really know anything about the field)

Re: The State of Probabilistic Programming

#13

Post by clementine »

nmrm2 wrote: post #3

"Having been rescued from the clutches of object-orientation by early exposure to Mathematica" I kind of laughed at this line, because this seems like a situation where the author came to the correct conclusion for all the wrong reasons. Maybe things have changed, but the Mathematica programming language never struck me as useful for anything over a couple thousand lines. edit: this is completely beside the purpose a…

Mathematica itself and Wolfram | Alpha have quite substantial code-bases written in Mathematica (now the Wolfram Language). Also, there was an article recently about probabilistic programming with WL:

http://www.mathematica-journal.com/data/uploads/2013/11/Cass...

My guess is that WL would be an interesting and powerful starting point for probabilistic programming at some point down the line (if not already).

Re: The State of Probabilistic Programming

#14

Post by cf »

detaro wrote: post #12
cf wrote: post #11

What's interesting about most complaints of these systems is people talk about their poor performance or scalability? That is usually more a consequence of using MCMC or other inference algorithm than the language itself. MCMC is a very slow inference algorithm. Its primary advantage was that for well-known models it could be coded up much more simply than a fancier inference technique. When you consider variational…

> The point of probabilistic programming is write inference algorithms once for a large class of models and be done. So the advantage of using a fancier method is amplified. So that should be something where a few standard libraries or toolsets should emerge that push out the "easy-to-implement" default choice? Are there any contenders yet? (As you might be able to tell, I don't really know anything about the field)

I know STAN and Figaro are going to push out inference methods like I mention, but my hope is eventually all of the ones mentioned in the article do this. I like thinking of this in terms of the standard library that needs to be built out. All the systems are making great progress in this regard.

Re: The State of Probabilistic Programming

#16

Post by ibab »

pjungwir wrote: post #4

This strikes me as a perfect application for quantum computers---but I'm just an amateur, so I'd love to hear an expert opinion. My understanding is this 1982 talk by Feynman [1] more or less launched the study of quantum computers, and it's all about how they can carry a probabilistic value through their computations rather than a definite one. And one of the lessons from that paper (if I'm reading & remembering rig…

You have to take into account that quantum processes are governed by a kind of "randomness" that is different from the one in stochastic processes. Scott Aaronson explains the basics of quantum mechanics very succinctly in one of his lectures [1].

The exponential->polynomial speedup we might get from a quantum computer doesn't really apply to stochastic processes, because we can already execute them in polynomial time on a classical computer (just use a random number generator).

Having a quantum computer would still be truly amazing, though. Even just being able to simulate quantum mechanics efficiently would be revolutionary.

[1] - http://www.scottaaronson.com/democritus/lec9.html

Re: The State of Probabilistic Programming

#17

Post by stdbrouw »

mizzao wrote: post #6

I've tried to use probabilistic programming for building a model of real data, and it seems that there is a long way to go before it's practical and fast. On one hand, there are the Monte Carlo-based methods that will support modeling almost any distributions, but are slow to use for large amounts of data. On the other hand, there are interesting cases like Infer.NET that use a completely different technique (approxi…

Well, MCMC is pretty close to brute force so it's never going to be very fast.

But the diagnostic tools are really quite good these days – tools like Stan and PyMC make it easy to see diagnostic plots that allow you to check for convergence and such. Additionally, more and more fine-tuning is happening automatically, like setting the right jump sizes and a plausible initial value for the random walk.

Note that traditional statistics, like regression, requires similar diagnostic steps: qqplots to check for the normality of your data, the Levene test for heteroskedasticity and so on. They often get ignored, but they're generally not hard to teach or learn.

We're really not that far from having this stuff be usable by the broader public. Maybe not "build any model you like and it will just work out of the box" but definitely as a replacement for much frequentist statistics. See "Doing Bayesian Data Analysis" by Kruschke, for example.

Re: The State of Probabilistic Programming

#20

Post by x0x0 »

stdbrouw wrote: post #17
mizzao wrote: post #6

I've tried to use probabilistic programming for building a model of real data, and it seems that there is a long way to go before it's practical and fast. On one hand, there are the Monte Carlo-based methods that will support modeling almost any distributions, but are slow to use for large amounts of data. On the other hand, there are interesting cases like Infer.NET that use a completely different technique (approxi…

Well, MCMC is pretty close to brute force so it's never going to be very fast. But the diagnostic tools are really quite good these days – tools like Stan and PyMC make it easy to see diagnostic plots that allow you to check for convergence and such. Additionally, more and more fine-tuning is happening automatically, like setting the right jump sizes and a plausible initial value for the random walk. Note that tradit…

I'm more pessimistic. My experience has been that inferring often benefits from reparameterization or analytic marginalizing, plus a bag of tricks that many people don't know.

It's unfortunately still a long way from being a technology that you can pretty blindly use like ols or logistic regression. Particularly if you only care about prediction, not parameter inference, lr with a penalty search is pretty straightforward.

May I ask what / if you use bayesian stats for professionally in your news work?

Also, the link to your github from your site goes to stdbrouw instead of debrouwere

Post reply on HN