Bayes’ Theorem – What is it and what is it good for?
crucialconsiderations.org
Bayes’ Theorem – What is it and what is it good for?
1–10 of 35 posts
Re: Bayes’ Theorem – What is it and what is it good for?
#2I'm in the middle of designing and building a system which uses Bayesian models.
One thing that struck me early is that while Bayes itself is rock solid, like arithmetic, when you go to apply it the results live or die on the quality of the models, and the relevance/realism of the evidence used to train them. GIGO.
But once you do have a good, relevant, signal-producing model, then, using it is a bit like doing a multi-dimensional lookup, or function call. Conceptually easy to understand, and, in many cases (depending, of course, on the details) cache-friendly.
Re: Bayes’ Theorem – What is it and what is it good for?
#3My biggest issue with Bayes' Theorem as a method of making everyday decisions is that it assumes the ability to accurately assess the underlying likelihoods of events taking place, especially on-the-fly.
I would even argue that it's actually providing a false sense of precision because the sig figs are oftentimes not correctly represented.
Re: Bayes’ Theorem – What is it and what is it good for?
#4Bayes' Theorem tells us that the quest for certain knowledge, which drove a great deal of science and philosophy in the pre-Bayesian era (before about 1990, when Bayesian methods started to gain real traction in the scientific community) is much like the alchemist's quest for the secret of transmutation: it is simply the wrong goal to have, even though it generated a lot of interesting and useful results.
One of the most important consequences of this is noted by the article: "Confirmation and falsification are not fundamentally different, as Popper argued, but both just special cases of Bayes’ Theorem." There is no certainty, even in the case of falsification, because there are always alternatives. For example, superluminal neutrinos didn't prove special relativity false, although they did provide some evidence. But the alternative hypothesis that the researchers had made a mistake turned out to be much more plausible.
Bayesian reasoning--which is plausibly the only way of reasoning that will keep our beliefs consistent with the evidence--cannot produce certainty. A certain belief is one that has a plausibility of exactly 1 or 0, and those are only asymptotically approachable applying Bayes' rule. Such beliefs would be immune from any further evidence for or against them, no matter how certain it was, essentially because Bayesian updating is multiplicative and anything times zero is still zero.
There is a name for beliefs of this kind, which to a Bayesian are the most fundamental kind of error: faith.
Re: Bayes’ Theorem – What is it and what is it good for?
#5My biggest issue with Bayes' Theorem as a method of making everyday decisions is that it assumes the ability to accurately assess the underlying likelihoods of events taking place, especially on-the-fly. I would even argue that it's actually providing a false sense of precision because the sig figs are oftentimes not correctly represented.
This is not a problem with Bayes' Theorem. Any alternative method of updating beliefs will suffer from exactly the same problem of noisy inputs, and have the the additional problem that it cannot maintain consistency with all the evidence (which only Bayes' rule is capable of doing.)
Using Bayes' rule consistently will make you aware of how uncertain the inputs are, and that is a feature, not a bug.
Re: Bayes’ Theorem – What is it and what is it good for?
#6My biggest issue with Bayes' Theorem as a method of making everyday decisions is that it assumes the ability to accurately assess the underlying likelihoods of events taking place, especially on-the-fly. I would even argue that it's actually providing a false sense of precision because the sig figs are oftentimes not correctly represented.
This is not a problem with Bayes' Theorem. Any alternative method of updating beliefs will suffer from exactly the same problem of noisy inputs, and have the the additional problem that it cannot maintain consistency with all the evidence (which only Bayes' rule is capable of doing.) Using Bayes' rule consistently will make you aware of how uncertain the inputs are, and that is a feature, not a bug.
How will it make you aware of uncertainty? At some point you do have to guess (called "estimating" here), do you not, and that will have a compounding effect on the outcome. You incorrectly guess a probability somewhere only by a small amount, and it multiplies its way through to the result, and you're looking at a potentially huge difference in resulting probability, which could easily span the "will act" or "won't act" gap.
Re: Bayes’ Theorem – What is it and what is it good for?
#7Bayes' Theorem tells us that the quest for certain knowledge, which drove a great deal of science and philosophy in the pre-Bayesian era (before about 1990, when Bayesian methods started to gain real traction in the scientific community) is much like the alchemist's quest for the secret of transmutation: it is simply the wrong goal to have, even though it generated a lot of interesting and useful results. One of the…
> Bayesian reasoning--which is plausibly the only way of reasoning that will keep our beliefs consistent with the evidence--cannot produce certainty.
To nitpick: Bayesian updating can produce certainty, in exactly the way you suggest: multiplying by zero. If the evidence you observed has zero probability under a particular hypothesis, then the posterior probability of that hypothesis will be zero. If the evidence you observe has zero probability under all hypotheses except for one, then posterior will give probability 1 to that hypothesis (assuming it had nonzero prior probability).
This won't come up if you're stick to densities like Gaussians that are supported everywhere. And it's certainly a good principle of model design to always allow your beliefs to be changed by new evidence (consistency theorems for Bayesian inference do depend on assumptions about the support of the prior and likelihood). But there's nothing formally preventing you from designing Bayesian models that rule out hypotheses with total certainty. In fact, this is what allows classical logic to be a special case of Bayesian reasoning.
Re: Bayes’ Theorem – What is it and what is it good for?
#8Bayes' Theorem tells us that the quest for certain knowledge, which drove a great deal of science and philosophy in the pre-Bayesian era (before about 1990, when Bayesian methods started to gain real traction in the scientific community) is much like the alchemist's quest for the secret of transmutation: it is simply the wrong goal to have, even though it generated a lot of interesting and useful results. One of the…
Bayes is perfect but also painfully bloodily sharp-edged. Best description I have.
Re: Bayes’ Theorem – What is it and what is it good for?
#9Bayes' Theorem tells us that the quest for certain knowledge, which drove a great deal of science and philosophy in the pre-Bayesian era (before about 1990, when Bayesian methods started to gain real traction in the scientific community) is much like the alchemist's quest for the secret of transmutation: it is simply the wrong goal to have, even though it generated a lot of interesting and useful results. One of the…
I awake. It is dark.
Therefore when I awake it is always dark.
Problem. Mismatch.
Turns out that if I awake between the local hours of 5am and 7pm then it is light. Otherwise it is dark. Problem. Mismatch. Turns out, it depends on the "time zone". Also turns out, depends on whether I'm sleeping inside or outside. In a hotel room or tent. Whether in a tent or in a building room with blinds. Etc. Etc. Each devil-in-the-details helps refine the case even further. But the "bet" to make is always the most "correct" bet to make, based only on the evidence observed to date, at hand. Thus Bayes.
Thus the Turing award.
It's just as perfect and reliable as that. And just as imperfect or vulnerable as that.
Re: Bayes’ Theorem – What is it and what is it good for?
#10Earlier quoted context omitted.
This is not a problem with Bayes' Theorem. Any alternative method of updating beliefs will suffer from exactly the same problem of noisy inputs, and have the the additional problem that it cannot maintain consistency with all the evidence (which only Bayes' rule is capable of doing.) Using Bayes' rule consistently will make you aware of how uncertain the inputs are, and that is a feature, not a bug.
How will it make you aware of uncertainty? At some point you do have to guess (called "estimating" here), do you not, and that will have a compounding effect on the outcome. You incorrectly guess a probability somewhere only by a small amount, and it multiplies its way through to the result, and you're looking at a potentially huge difference in resulting probability, which could easily span the "will act" or "won't…
I think it's worth keeping the principles of bayesian reasoning in mind. The idea that you should update hypotheses on evidence, and adding probability in one place takes it from everywhere else slightly, keeping track of prior probability, etc.
Not that you should actually do mental calculations on made up probability estimates. I mean you can do that, and if your estimates are at all decent, the result might be better. But I don't think anyone actually recommends that.