Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bayes’ Theorem in the 21st Century (2013) [pdf]

web.ipac.caltech.edu

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#81

Post by abdullahkhalids »

bo1024 wrote: post #45
xcodevn wrote: post #16

Earlier quoted context omitted.

I strongly disagree. Bayesian inference is the only known self-consistent formal system for doing science, i.e. updating our belief system about the world based on the current evidence.

The problem with that reasoning is: whose belief system? Where do you come up with a prior that everyone agrees with?

You are not supposed to agree on a prior. That's one of the fundamental insights of the Bayesian inference framework. That different people know different things about a given situation, so they initially disagree, and therefore their priors are different. This should not be surprised. People disagree all the time, and the Bayesian framework just formalizes it.

The different people can then go on and do lots of experiments, collect lots of data and update their priors to posteriors. And the guarantee is that as long as each person's prior was not a mathematically weird function, after enough evidence has been collected all these people will have the same posterior function i.e. they will agree [1].

[1] The famous Aumann's agreement theorem https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem is a related result that you might like to read about.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#82

Post by bo1024 »

KirinDave wrote: post #79
bo1024 wrote: post #45

Earlier quoted context omitted.

The problem with that reasoning is: whose belief system? Where do you come up with a prior that everyone agrees with?

Actually, it's the strength of Bayesian inference that these assumptions are made apparent. Coming to consensus on priors is the same process for arriving at consensus that all scientific inquiry must engage in. Anyone who says frequentist methods somehow more accurately represent an underlying reality are pulling a fast one.

Hmm, I think part of the question is where this debate and consensus should occur. I believe in firmly separating rigorous science from opinion and belief. To me, it follows that scientific research should focus on presenting evidence and leave it to Bayesian individuals to update their beliefs based on this evidence. Similarly I think argument or discussion about priors is not in scope for scientific research (except maybe a bit in the "motivation" subsection). (edit: I see now the top post here on this article also discusses this point.)

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#83

Post by bo1024 »

abdullahkhalids wrote: post #81
bo1024 wrote: post #45

Earlier quoted context omitted.

The problem with that reasoning is: whose belief system? Where do you come up with a prior that everyone agrees with?

You are not supposed to agree on a prior. That's one of the fundamental insights of the Bayesian inference framework. That different people know different things about a given situation, so they initially disagree, and therefore their priors are different. This should not be surprised. People disagree all the time, and the Bayesian framework just formalizes it. The different people can then go on and do lots of exper…

Exactly. This is why (in my view) scientific research should focus on presenting evidence, not on arguing for certain posteriors or priors. The meta-science process then steers Bayesian beliefs correctly and the evidence-gathering process efficiently. (edit: I see now the top post here on this article also discusses this point.)

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#84

Post by xcodevn »

mturmon wrote: post #77
xcodevn wrote: post #29

Earlier quoted context omitted.

> It is very hard to validate a given choice of a prior in many applications. E.g., if I claim one prior, and another investigator claims a sharper one, it can be very difficult to decide who is right. Both prior and likelihood are our model's assumptions. So, the prior validation problem is similar to the likelihood validation problem. To check a Bayesian model or any model, we need to bring the model out of the for…

> ...the prior validation problem is similar to the likelihood validation problem... But priors can be much harder. Say I’m trying to estimate a wind speed from the blade velocity of a windmill. I can bring a more accurate wind speed sensor to calibrate the windmill against the wind speed, perhaps aided by basic physics. This is the likelihood portion. But what should the prior be? The typical speed at that time of d…

> I can bring a more accurate wind speed sensor to calibrate the windmill against the wind [...] But what should the prior be? [...] I have to have a crisp number — a full distribution actually, accurate out to the tails.

What would you do when the sensor returns negative wind speeds due to noise or errors?

The wind speed cannot be negative, or greater than speed of light. An expert in windmill can narrow down the prior distribution much more.

> But priors can be much harder.

Choosing a prior is hard because it requires thinking explicitly about the problem and its assumptions. It merely exposes our lack of expertise on the problem.

When you're lazy, you can pick a uniform prior Uniform(0, c) and call it a day.

> We can sharpen the problem. Say my number and yours are different. How do we tell who is right?

Forget about prior, say, we have 2 sensors which output two slightly different wind speeds. Which wind speeds is right? The lower one or the average speed.

This is a deep philosophical problem. However, it's a problem for any model.

> The prior will mostly wash out in any well-posed problem.

I don't think so. Any well-posed problem should include the prior, or else how can we tell: 2 data points is not enough?

> HN tends to invoke the Bayesian framework as a complete solution to inference [...]

Bayesian framework is indeed a complete solution to inference in a formal/logical sense. However, I agree that there are many problems in applying Bayesian framework to real world problems that requires serious thinking about our assumptions on the problem.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#85

Post by meatmanek »

platz wrote: post #26
xcodevn wrote: post #2

I truly believe that Bayesian inference is the statistics of the 21st century. Recent advances in MCMC (e.g., NUTS, Stan [1]) and variational inference (e.g., ADVI [2], VAE [3], etc.) + more computing power than ever promise a near future in which Bayesian inference is the default inference engine. Prior distribution is a beautiful and logical mechanism for adding regularization, domain-specific knowledge to our mode…

I feel like variational inference has never been described very well to an intro audience even having statistical basics. Is it a graduate level topic or is there an intuitive course that teaches it to beginners?

I found this talk to be useful, despite the technical difficulties https://www.youtube.com/watch?v=Dv86zdWjJKQ

(I haven't watched https://www.youtube.com/watch?v=ogdv_6dbvVQ but it seems like a longer version of the same talk)

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#86

Post by paulgrant999 »

13415 wrote: post #21
Vaslo wrote: post #12

One of the best examples of the idea of Bayes is the Monte Hall problem. It is a good example of how prior probability (3 unopened doors) can lead to a more clear posterior-host selects an unopened door with a bad prize and you are asked whether to stay with the unopened door you chose or switch to the remaining open door. Turns out via Bayes it’s better to switch doors because you have more information now. Tons of…

It's better to switch doors according to any statistical method, whether you're a frequentist or Bayesian does not matter. You can also show that you should switch doors by making an exhaustive truth table, by writing a computer program, or by experimentation, if you prefer these kind of approaches.

I made this very same point ;) they burned me at the stake.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#88

Post by Koshkin »

akvadrako wrote: post #74
Koshkin wrote: post #71

Earlier quoted context omitted.

> The Frequentist believes The frequentists do not "believe," they measure .

No. They believe you can measure an infinite number of trials (say # of heads vs tails) and whatever ratio you get is the probability of heads. However it's problematic because you can measure a million coin flips and get heads every time. It's not possible to actually measure an infinite number of trials - you need to imagine it.

This is just silly, nobody in their right mind would believe they could do something “an infinite number” of times.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#89

Post by Koshkin »

platz wrote: post #72
Koshkin wrote: post #71

Earlier quoted context omitted.

> The Frequentist believes The frequentists do not "believe," they measure .

They still non-trivially define/demarcate what the population actually is. That is kind of a belief because it is a choice not given by nature, and there are infinitely many choices one could choose.

Nature does not give you choice, it gives you the frequency (say, in the form of the intensity of a spectral line of an atom), and it is the base of the scientific method to listen to what nature is trying to tell you. There is nothing subjective in this process. Arguing otherwise is like saying that atheists “believe” in the non-existence of god.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#90

Post by akvadrako »

Koshkin wrote: post #88
akvadrako wrote: post #74

Earlier quoted context omitted.

No. They believe you can measure an infinite number of trials (say # of heads vs tails) and whatever ratio you get is the probability of heads. However it's problematic because you can measure a million coin flips and get heads every time. It's not possible to actually measure an infinite number of trials - you need to imagine it.

This is just silly, nobody in their right mind would believe they could do something “an infinite number” of times.

If you don’t do an infinite number of trials then you can’t be sure your frequencies match the real probability.
Post reply on HN