Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bayes’ Theorem in the 21st Century (2013) [pdf]

web.ipac.caltech.edu

Post reply on HN
Print view View original

71–80 of 97 posts

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#71

Post by Koshkin »

platz wrote: post #57
loup-vaillant wrote: post #46

http://www.overcomingbias.com/2009/02/share-likelihood-ratio... Seriously, the main point of an experiment is to gather evidence . Coupled with prior beliefs, you get a posterior belief, but the most important point is how much evidence the experiment provides. Sure, a full fledged posterior belief is needed to make an actual decision, like, what should we test next. And if a subject is deemed important enough that w…

The Bayesian believes that probability represents our beliefs about the world. The Frequentist believes that probabilities merely represent the long term frequency counts of events (for a given 'population').

> The Frequentist believes

The frequentists do not "believe," they measure.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#72

Post by platz »

Koshkin wrote: post #71
platz wrote: post #57

Earlier quoted context omitted.

The Bayesian believes that probability represents our beliefs about the world. The Frequentist believes that probabilities merely represent the long term frequency counts of events (for a given 'population').

> The Frequentist believes The frequentists do not "believe," they measure .

They still non-trivially define/demarcate what the population actually is. That is kind of a belief because it is a choice not given by nature, and there are infinitely many choices one could choose.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#73

Post by ced »

klipt wrote: post #65
platz wrote: post #26

Earlier quoted context omitted.

I feel like variational inference has never been described very well to an intro audience even having statistical basics. Is it a graduate level topic or is there an intuitive course that teaches it to beginners?

"variational inference" is perhaps an uninformative name. You can just think of it as - approximating the posterior using a nice parametric distribution, then - minimizing some error (typically KL Divergence) between your approximate posterior and the true posterior

Do you know _why_ KL divergence is minimized? I get that it gives a lower bound on the marginal likelihood, which is cool, but is that it? What are the alternatives?

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#74

Post by akvadrako »

Koshkin wrote: post #71
platz wrote: post #57

Earlier quoted context omitted.

The Bayesian believes that probability represents our beliefs about the world. The Frequentist believes that probabilities merely represent the long term frequency counts of events (for a given 'population').

> The Frequentist believes The frequentists do not "believe," they measure .

No. They believe you can measure an infinite number of trials (say # of heads vs tails) and whatever ratio you get is the probability of heads.

However it's problematic because you can measure a million coin flips and get heads every time. It's not possible to actually measure an infinite number of trials - you need to imagine it.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#75

Post by ssivark »

ced wrote: post #73
klipt wrote: post #65

Earlier quoted context omitted.

"variational inference" is perhaps an uninformative name. You can just think of it as - approximating the posterior using a nice parametric distribution, then - minimizing some error (typically KL Divergence) between your approximate posterior and the true posterior

Do you know _why_ KL divergence is minimized? I get that it gives a lower bound on the marginal likelihood, which is cool, but is that it? What are the alternatives?

KL divergence is motivated nicely from an information/coding theory viewpoint. It's very closely related to Shannon-von Neumann entropy [1], and KL(P||Q) characterizes the efficiency of a code designed for a model distribution P, when applied to reality which is actually represented by Q.

A lot of recent work focuses on the Wasserstein divergence [1] as an alternative. One advantage of Wasserstein over KL is that the Wasserstein metric provides better fit over the whole distribution instead of localizing on some specific regions, thereby preventing "mode collapse". This makes it a popular metric for training Generative Adversarial Networks (GANs).

For recent work on applying Wasserstein distance to variational inference, see: https://arxiv.org/abs/1805.11284

[1]: https://physics.stackexchange.com/questions/64574/definition... [2]: https://en.wikipedia.org/wiki/Wasserstein_metric

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#77

Post by mturmon »

xcodevn wrote: post #29
mturmon wrote: post #24

Earlier quoted context omitted.

But note, the last paragraph of TFA cautions against use of a Bayesian prior in cases where it is not well supported by actual hard prior information. It is very hard to validate a given choice of a prior in many applications. E.g., if I claim one prior, and another investigator claims a sharper one, it can be very difficult to decide who is right. If the prior does not wash out due to lots of data, this indicates a…

> It is very hard to validate a given choice of a prior in many applications. E.g., if I claim one prior, and another investigator claims a sharper one, it can be very difficult to decide who is right. Both prior and likelihood are our model's assumptions. So, the prior validation problem is similar to the likelihood validation problem. To check a Bayesian model or any model, we need to bring the model out of the for…

> ...the prior validation problem is similar to the likelihood validation problem...

But priors can be much harder.

Say I’m trying to estimate a wind speed from the blade velocity of a windmill. I can bring a more accurate wind speed sensor to calibrate the windmill against the wind speed, perhaps aided by basic physics. This is the likelihood portion.

But what should the prior be? The typical speed at that time of day? The speed in January? The speed on cloudy days? I have to have a crisp number — a full distribution actually, accurate out to the tails. I really have very little grounding for choosing that distribution.

I started out just wanting to relate the wind speed to some data in a rather concrete way, and now I’ve been roped in to choosing a crisp distribution for a rather amorphous state of nature.

This is a deep problem.

We can sharpen the problem. Say my number and yours are different. How do we tell who is right?

One can try a different tack: I’m being stubborn. The prior will mostly wash out in any well-posed problem, or else why try to solve it? But now we’re back to frequentism, just looking at the likelihood.

HN tends to invoke the Bayesian framework as a complete solution to inference — I’m just trying to demonstrate that there are problems with that approach.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#78

Post by KirinDave »

mturmon wrote: post #77
xcodevn wrote: post #29

Earlier quoted context omitted.

> It is very hard to validate a given choice of a prior in many applications. E.g., if I claim one prior, and another investigator claims a sharper one, it can be very difficult to decide who is right. Both prior and likelihood are our model's assumptions. So, the prior validation problem is similar to the likelihood validation problem. To check a Bayesian model or any model, we need to bring the model out of the for…

> ...the prior validation problem is similar to the likelihood validation problem... But priors can be much harder. Say I’m trying to estimate a wind speed from the blade velocity of a windmill. I can bring a more accurate wind speed sensor to calibrate the windmill against the wind speed, perhaps aided by basic physics. This is the likelihood portion. But what should the prior be? The typical speed at that time of d…

To me, and I'm not quite the expert that perhaps you are, but to me it seems like Bayesian inference is still in a better spot here because the priors are part of an explicit quantification of bias and assumption in a model.

Much havoc has befallen the scientific world because of the hidden assumptions of frequentist techniques with poorly understood preconditions, even for rather basic models. And there isn't much anyone can do about that save move to ever more complicated models.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#79

Post by KirinDave »

bo1024 wrote: post #45
xcodevn wrote: post #16

Earlier quoted context omitted.

I strongly disagree. Bayesian inference is the only known self-consistent formal system for doing science, i.e. updating our belief system about the world based on the current evidence.

The problem with that reasoning is: whose belief system? Where do you come up with a prior that everyone agrees with?

Actually, it's the strength of Bayesian inference that these assumptions are made apparent.

Coming to consensus on priors is the same process for arriving at consensus that all scientific inquiry must engage in. Anyone who says frequentist methods somehow more accurately represent an underlying reality are pulling a fast one.

Re: Bayes’ Theorem in the 21st Century (2013) [pdf]

#80

Post by abdullahkhalids »

mturmon wrote: post #77
xcodevn wrote: post #29

Earlier quoted context omitted.

> It is very hard to validate a given choice of a prior in many applications. E.g., if I claim one prior, and another investigator claims a sharper one, it can be very difficult to decide who is right. Both prior and likelihood are our model's assumptions. So, the prior validation problem is similar to the likelihood validation problem. To check a Bayesian model or any model, we need to bring the model out of the for…

> ...the prior validation problem is similar to the likelihood validation problem... But priors can be much harder. Say I’m trying to estimate a wind speed from the blade velocity of a windmill. I can bring a more accurate wind speed sensor to calibrate the windmill against the wind speed, perhaps aided by basic physics. This is the likelihood portion. But what should the prior be? The typical speed at that time of d…

My perspective is that the problem of deciding the "correct" prior is a human problem because the human brain is a messy machine. An artificial intelligence which has full access to its own code and its memory in perfect detail will know precisely what it knows about a certain situation, and therefore can estimate a prior that accurately reflects this knowledge.

In the windmill example, the AI can quickly collect all it has in its memory about blade speeds, and maybe spend a self-imposed X min computational time to make a best guess for the prior speed distribution.

Humans can't do this, so we have gone down a philosophical rabbit hole of figuring out this "prior problem", when the real problem is that we are just messy informal thinkers.

> How do we tell who is right?

You are fundamentally conceptually mistaken here. There is nothing right or wrong with two agents disagreeing on the prior. The different prior reflects the before experiment knowledge of the two agents. I am a windmill engineer, so my priors will be much more narrow than yours, who has never seen a windmill outside of a hollywood movie.

Post reply on HN