Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Probabilistic Programming & Bayesian Methods for Hackers

camdavidsonpilon.github.io

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Probabilistic Programming & Bayesian Methods for Hackers

#11

Post by darkxanthos »

xmpir wrote: post #10

I also like "Think Bayes" even though it is "just" a book. http://www.greenteapress.com/thinkbayes/

I've been working with both books and Think Bayes is more accessible (and its also free). I recommend going through it before getting to PyMC.

One huge reason why is that the author has implemented everything you use in Python in a way that enables you to read his code to more fully understand what's happening. I'm on chapter 5 or 6 and he hasn't even touched on MCMC yet which is most welcome.

Re: Probabilistic Programming & Bayesian Methods for Hackers

#13

Post by martincmartin »

For those who want a more solid take on machine learning, and who still remember their math and probability/statistics, (i.e. advanced undergrad or new grad student), the best texts seem to be:

The Elements of Statistical Learning by Hastie, Tibshirani and Friedman, available for free on line.

Pattern Recognition and Machine Learning by Chris Bishop. Very Bayesian.

Machine Learning: A Probabilistic Perspective by Kevin Murphy. Also Bayesian, although not as Bayesian as Bishop. The most recent of the three, and therefore covers a few topics not covered elsewhere like deep learning and conditional random fields. The first few printings are full of errors and confusing passages, should be better before too long.

Did I miss any?

Re: Probabilistic Programming & Bayesian Methods for Hackers

#14

Post by gtani »

martincmartin wrote: post #13

For those who want a more solid take on machine learning, and who still remember their math and probability/statistics, (i.e. advanced undergrad or new grad student), the best texts seem to be: The Elements of Statistical Learning by Hastie, Tibshirani and Friedman, available for free on line. Pattern Recognition and Machine Learning by Chris Bishop. Very Bayesian. Machine Learning: A Probabilistic Perspective by Kev…

There's quite a few good free ones of varying perspectives

http://metaoptimize.com/qa/questions/186/good-freely-availab...

Esp:

http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=...

http://www.inference.phy.cam.ac.uk/itprnn/book.html

http://alex.smola.org/drafts/thebook.pdf

___________

and MLPP errata: http://www.cs.ubc.ca/~murphyk/MLbook/errata.html

Re: Probabilistic Programming & Bayesian Methods for Hackers

#15

Post by Blahah »

martincmartin wrote: post #13

For those who want a more solid take on machine learning, and who still remember their math and probability/statistics, (i.e. advanced undergrad or new grad student), the best texts seem to be: The Elements of Statistical Learning by Hastie, Tibshirani and Friedman, available for free on line. Pattern Recognition and Machine Learning by Chris Bishop. Very Bayesian. Machine Learning: A Probabilistic Perspective by Kev…

I would add to that two classics with free PDFs available from the authors' websites:

1. David Barber: Bayesian Reasoning and Machine Learning

2. David MacKay: Information Theory, Inference, and Learning Algorithms

[1] http://www.cs.ucl.ac.uk/staff/d.barber/brml/

[2] http://www.inference.phy.cam.ac.uk/mackay/itila/

Both require a willingness to immerse in the mathematics. But the maths isn't hard, even for someone like me who hasn't formally studied it since I was 15.

Re: Probabilistic Programming & Bayesian Methods for Hackers

#16

Post by ogrisel »

imurray wrote: post #8
jkldotio wrote: post #5

Earlier quoted context omitted.

I don't see any confusion at all. The first few paragraphs of the link say it's based on PyMC, which itself appears in your link under "Existing probabilistic programming systems". So it's a book that's a practical guide to using one of the systems you reference.

As a researcher with MCMC interests, I agree with the grandparent post. Used as technical jargon "probabilistic programming" tends to mean: specify a model using a programming language, a compiler then works out how to do inference in that model, and writes the inference code for you. PyMC is a toolkit that makes it easier to write inference code for a wide range of models, but isn't as automatic as the field of prob…

PyMC follows the library approach to probabilistic programming rather than inventing yet another application specific language that only a niche of developers will be willing to spend time to learn.

Despite not introducing a new language syntax or DSL, PyMC is still probabilistic programming in the sense that you have python variables that represent random variables with prior distributions and then use those to derive new random variables by using deterministic python expressions or functions. Finally you plug an inference engine to be able to invert the execution order and derive a posterior distribution on the unobserved variables.

Re: Probabilistic Programming & Bayesian Methods for Hackers

#17

Post by tristanz »

ogrisel wrote: post #16
imurray wrote: post #8

Earlier quoted context omitted.

As a researcher with MCMC interests, I agree with the grandparent post. Used as technical jargon "probabilistic programming" tends to mean: specify a model using a programming language, a compiler then works out how to do inference in that model, and writes the inference code for you. PyMC is a toolkit that makes it easier to write inference code for a wide range of models, but isn't as automatic as the field of prob…

PyMC follows the library approach to probabilistic programming rather than inventing yet another application specific language that only a niche of developers will be willing to spend time to learn. Despite not introducing a new language syntax or DSL, PyMC is still probabilistic programming in the sense that you have python variables that represent random variables with prior distributions and then use those to deri…

I tend to agree with Ian that it's confusing to conflate probabilistic programming and libraries that support Bayesian inference. In PyMC variables represent random variables, but these variables can't be used with Python constructs like conditionals and loops. Python is used to construct a DAG, which is then executed.

I think a better definition of probabilistic programming languages are languages where you can replace any variable of type T with Random. The line isn't entirely clear, but library approaches in languages like Python don't fit, since they can't handle control flow. Bugs/JAGS/Stan might qualify, although they are very limited declarative languages. Their motivation is primarily to have a compact modeling syntax, not a real programming language.

There's no need for probabilistic programming languages to be some esoteric DSL. You can convert languages like Python or Matlab into a probablistic programming language with a lightweight compiler transformation: http://www.mit.edu/~wingated/papers/lightweight_pp.pdf. Actually doing inference efficiently, however, remains as challenging as always.

Re: Probabilistic Programming & Bayesian Methods for Hackers

#18

Post by stiff »

martincmartin wrote: post #13

For those who want a more solid take on machine learning, and who still remember their math and probability/statistics, (i.e. advanced undergrad or new grad student), the best texts seem to be: The Elements of Statistical Learning by Hastie, Tibshirani and Friedman, available for free on line. Pattern Recognition and Machine Learning by Chris Bishop. Very Bayesian. Machine Learning: A Probabilistic Perspective by Kev…

The best introductory book and also the most cohesive one is "Learning from Data" from Yaser Abu-Mostafa, accompanied by great video lectures:

http://amlbook.com/

It differs from other books in that all the material is treated from the unified perspective of statistical learning theory and VC dimension, as a result the book feels less like a hodgepodge of unrelated techniques and more like an introduction to a coherent field.

Hastie and Tibshirani also have a new, less demanding mathematically book out:

http://www.amazon.com/Introduction-Statistical-Learning-Appl...

Re: Probabilistic Programming & Bayesian Methods for Hackers

#19

Post by baudehlo »

martincmartin wrote: post #13

For those who want a more solid take on machine learning, and who still remember their math and probability/statistics, (i.e. advanced undergrad or new grad student), the best texts seem to be: The Elements of Statistical Learning by Hastie, Tibshirani and Friedman, available for free on line. Pattern Recognition and Machine Learning by Chris Bishop. Very Bayesian. Machine Learning: A Probabilistic Perspective by Kev…

Pg's own A Plan for Spam is also an excellent introduction to Bayes for boolean comparison.

http://www.paulgraham.com/spam.html

Re: Probabilistic Programming & Bayesian Methods for Hackers

#20

Post by jkldotio »

tristanz wrote: post #17
ogrisel wrote: post #16

Earlier quoted context omitted.

PyMC follows the library approach to probabilistic programming rather than inventing yet another application specific language that only a niche of developers will be willing to spend time to learn. Despite not introducing a new language syntax or DSL, PyMC is still probabilistic programming in the sense that you have python variables that represent random variables with prior distributions and then use those to deri…

I tend to agree with Ian that it's confusing to conflate probabilistic programming and libraries that support Bayesian inference. In PyMC variables represent random variables, but these variables can't be used with Python constructs like conditionals and loops. Python is used to construct a DAG, which is then executed. I think a better definition of probabilistic programming languages are languages where you can repl…

>it's confusing to conflate probabilistic programming and libraries that support Bayesian inference

But it's a generic term so you could say the same about functional programming or logic programming, both of which can be done in Python even if there are more advanced or integrated systems elsewhere. I don't really think most people care, besides perhaps PL researchers, at which portion of the stack things are happening at or being optimised; if you are using the relevant mathematics and statistics that's what you are doing. I think people are playing semantics to say it only means one thing when it's obviously used in a general way and a sometimes in specific way.

The bottom line is the guy who wrote the book thinks it's probabilistic programming, ogrisel does, I do, and the people who run http://probabilistic-programming.org/wiki/Home seem to be referring to it as probabilistic programming as well. I don't buy Ian's argument that it's part of some latter type of category on the site, PyMC is directly linked in a section titled "Existing probabilistic programming systems". They use "as well as" to link the two groups so either the first is "systems" and the rest are still "probabilistic programming" just without "systems" or they are all "probabilistic programming systems" if "as well as" is operating in that way. The arguments against this seem to be splitting hairs and playing semantics far too much when n-grams regularly have more than one meaning. Indeed it's amusing to see probabilistic people arguing for one interpretation rather than saying that there could be more than one and it depends on context (an NLP program trying to disambiguate the meaning of a given n-gram would look at other words present, topic models for the document, et cetera).

Post reply on HN