How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy • Hacker News Board

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#61

Post by conjectures » Mon, Jun 27, 2016, 5:10 PM UTC

danbruc wrote: post #40

kareemsabri wrote: post #39

Earlier quoted context omitted.

Not at all! It yields that sequence all the time, in small numbers. But, it is impossible to get only heads, if you could truly flip a coin infinitely many times, which of course you can't. The larger you make the sequence, the smaller you make the probability of all heads. In your example: P(H,H,H,H) = 0.5 ^ 4 = 0.0625 P(H,H,H,H,H,H,H,H) = 0.5 ^ 8 = 0.00390625 As you can see it's getting pretty small already. Howeve…

But now you are contradicting yourself, aren't you? We agree that for any finite sequence of tosses it is unlikely but possible to get all heads and therefore it is possible to not converge to 0.5. The question is what difference it makes to go from a large but finite to an infinite number of tosses. Either it is impossible to get only heads an infinite number of times, then I have a problem understanding why that is…

When heads has probability p the probability for any n-length sequence of all heads is p^n. For p between 0 and 1 this vanishes as n goes to infinity.

The reason is that the conjunction of some events is always less (than or equal to) probable than the events themselves. This should make intuitive sense as the middle bit of a Venn diagram. Even more so for a conjunction of events with a conjunction of events. When taken to the infinite limit you tend to end up with the conjunction having probability 0 or 1.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#62

Post by kgwgk » Mon, Jun 27, 2016, 5:20 PM UTC

hacker42 wrote: post #19

I could never quite understand the divide between Bayesian statistics and frequentist statistics. Both seem to be ultimately about counting the frequency by which something occurs and normalizing this frequency with respect to the number of all possible outcomes. Bayesian statistics essentially is concerned with the application of the Bayesian updating technique by which one can iteratively improve a distribution ove…

Bayesian probability cannot always be interpreted as a frequency. For example, one could assign a bayesian probability to the extra-terrestrial origin of life. It wouldn't make much sense to think of it as a frequentist probability: one can easily imagine playing the future several times, but it's not so easy when dealing with the past.

And statistics is not just probability. Frequentist inference is based on procedures that "behave well" in the long term, but may or may not make sense for the particular outcome at hand.

For example, a 95% confidence interval calculated using a procedure that guarantees that the interval contains the true value 95% of the time may yield an interval that cannot contain the true value (for example the interval covers only negative values and the true value is known to be positive). See http://learnbayes.org/papers/confidenceIntervalsFallacy/ for a discussion of confidence intervals.

Another issue is related to how the "possible outcomes" are defined. For example, a frequentist analysis of the fairness of a coin after getting four heads and then a tail will be different depending on whether we decided to throw the coin until getting a tail or we had fixed beforehand the number of trials. Look for "stopping rules" or "optional stopping."

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#63

Post by GregBuchholz » Mon, Jun 27, 2016, 5:29 PM UTC

hacker42 wrote: post #19

I could never quite understand the divide between Bayesian statistics and frequentist statistics. Both seem to be ultimately about counting the frequency by which something occurs and normalizing this frequency with respect to the number of all possible outcomes. Bayesian statistics essentially is concerned with the application of the Bayesian updating technique by which one can iteratively improve a distribution ove…

Have you ever taken a look at: "Probability Theory: The Logic of Science"

http://www.med.mcgill.ca/epidemiology/hanley/bios601/Gaussia...

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#64

Post by kgwgk » Mon, Jun 27, 2016, 7:18 PM UTC

danbruc wrote: post #28

paol wrote: post #25

Earlier quoted context omitted.

> For any finite number of tosses you can certainly not converge to 0.5 I think you mean "may not" rather than "can not". But yes, that's why you need to resort to limits[0] to understand the problem. [0] https://en.wikipedia.org/wiki/Limit_of_a_sequence

Thanks, fixed. It understand that it is the limit, but the question I am asking is whether it is possible to always get heads, an infinite number of times. If this is possible, then it is not true that the relative frequency always approaches 0.5 as the number of repetitions goes to infinity. I understand that the probability of obtaining heads an infinite number of times is zero, but does that mean it is impossible?…

There are different ways to "approach 0.5 as the number of repetitions goes to infinity": https://en.wikipedia.org/wiki/Convergence_of_random_variable...

I think in this case you have "convergence in probability" (but I've not read carefully the discussion). There is a stronger form, "almost sure convergence", where there is exact convergence with probability one (i.e. in some cases there is no convergence, but those happen with probability zero).

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#65

Post by greenshackle » Mon, Jun 27, 2016, 7:29 PM UTC

dschiptsov wrote: post #55

dschiptsov wrote: post #3

Brexit is the best example so far. That painful dissonance between so called reality and these probabilistic models.

Probabilities makes sense only with absolutely certain things like a fair coin or a dice. In cases where there is no absolute certainty about how many sides or dimensions your "dice" has and that it is not biased and that there is no other forces or factors in play probability ceases to make sense. Probability of A, given B becomes meaningless when either A or B aren't precisely defined (like in the case of a "fair c…

I disagree. There's a large grey area between 'completely unknown' and 'scientific certainty'. I prefer guessing and doing computations with my guesses than throwing my hands up in the air and calling it unknowable.

When presented with new evidence it's better to write a number down for your degree in belief in X, ask how much you should change that belief based on the new data, and update your probability estimate, than to just go with your feelings. You're not doing actual bayesian computations, that's totally untractable for anything that's not a very well defined problem, but doing 'pseudo-bayesian' updating is better than not.

I think of it as the fermi approximation of probabilities. You won't get accurate numbers that way but you'll get better numbers than if you just invent the answer.

EDIT to add: most of the time you should then throw out the number. Just like a fermi estimate, you get a ballpark sense for the answer, not a precise answer.

In the superforecasting experiment by Tetlock the best forecasters did this. They were writing down probability estimates and methodically updating them based on new data (news articles, data, etc.). They were forecasting geopolitical events, not dice rolls, and it worked (better than the alternative, obviously no one can forecast geopolitical events with high certainty).

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#66

Post by hacker42 » Mon, Jun 27, 2016, 8:22 PM UTC

kgwgk wrote: post #62

hacker42 wrote: post #19

I could never quite understand the divide between Bayesian statistics and frequentist statistics. Both seem to be ultimately about counting the frequency by which something occurs and normalizing this frequency with respect to the number of all possible outcomes. Bayesian statistics essentially is concerned with the application of the Bayesian updating technique by which one can iteratively improve a distribution ove…

Bayesian probability cannot always be interpreted as a frequency. For example, one could assign a bayesian probability to the extra-terrestrial origin of life. It wouldn't make much sense to think of it as a frequentist probability: one can easily imagine playing the future several times, but it's not so easy when dealing with the past. And statistics is not just probability. Frequentist inference is based on procedu…

What would you respond to my other comment downthread?

https://news.ycombinator.com/item?id=11985863

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#67

Post by kgwgk » Mon, Jun 27, 2016, 8:39 PM UTC

hacker42 wrote: post #66

kgwgk wrote: post #62

Earlier quoted context omitted.

Bayesian probability cannot always be interpreted as a frequency. For example, one could assign a bayesian probability to the extra-terrestrial origin of life. It wouldn't make much sense to think of it as a frequentist probability: one can easily imagine playing the future several times, but it's not so easy when dealing with the past. And statistics is not just probability. Frequentist inference is based on procedu…

What would you respond to my other comment downthread? https://news.ycombinator.com/item?id=11985863

If you are able to think of the different realities that would lead to us having this discussion today, some of them with life being originated on planet Earth and some of them with life coming from elsewhere, and you're able to reason about the relative frequency of these two kinds of realities, you definitely have more imagination than me.

And I don't know what do you gain with that. Does your frequentist interpretation have a physical meaning? I could assign a Bayesian probability to panspermia, and you could assign a different probability. Is there, according to your equivalent definition in terms of frequencies, a "correct" probability?

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#68

Post by effie » Mon, Jun 27, 2016, 8:49 PM UTC

danbruc wrote: post #28

paol wrote: post #25

Earlier quoted context omitted.

> For any finite number of tosses you can certainly not converge to 0.5 I think you mean "may not" rather than "can not". But yes, that's why you need to resort to limits[0] to understand the problem. [0] https://en.wikipedia.org/wiki/Limit_of_a_sequence

Thanks, fixed. It understand that it is the limit, but the question I am asking is whether it is possible to always get heads, an infinite number of times. If this is possible, then it is not true that the relative frequency always approaches 0.5 as the number of repetitions goes to infinity. I understand that the probability of obtaining heads an infinite number of times is zero, but does that mean it is impossible?…

"whether it is possible to always get heads, an infinite number of times" No if we actually make the experiment, for we cannot toss coin infinitely many times. Yes, if we are just considering and measuring all possible infinite sequences. Measure of such mathematically possible sequence is zero, like measure of a point somewhere in a circle is zero. The point exists and is a possible result of choosing a point, but its probability is zero. Thus zero probability does not mean thing is impossible; it is only very unlikely.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#69

Post by danbruc » Mon, Jun 27, 2016, 9:17 PM UTC

kareemsabri wrote: post #52

danbruc wrote: post #47

Earlier quoted context omitted.

Okay, assuming that is true, is there an intuitive way to understand that? And the fact that lim[n-> ∞] 0.5^n = 0 unfortunately won't do for me, that is true for any specific infinite sequence, even those containing 50/50 heads and tails. I think most of the sequences - hand-waving, most of an infinite set - are 50/50 heads and tails just because there are more possibilities - hand-waving again - to arrange 50/50 hea…

> Okay, assuming that is true, is there an intuitive way to understand that? And the fact that lim[n-> ∞] 0.5^n = 0 unfortunately won't do for me, that is true for any specific infinite sequence, even those containing 50/50 heads and tails. That's true, but only if you pick ONE specific sequence. The likelihood of any precise sequence of heads/tails is equally unlikely as all heads. However, we are not fixing the out…

Well, we've already proven it's impossible to have all heads infinitely. The same would be true of any non-uniform sequence. Any sequence that favours heads by any margin at all, taken to infinity, would necessarily contain an infinite sub-sequence that contains all heads, which we've proven is impossible by the limit to infinity.

Have we? We only said that the probability of having an infinite sequence of all heads is zero but not that it is impossible to obtain that result. After all the probability of any infinite sequence is zero and therefore if we would equate probability zero with impossible to obtain, then we could obtain no infinite sequence at all. The second part is obviously wrong, H H T H H T... ad infinitum obviously favors heads over tails but does not contain an infinite subsequence of all heads.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#70

Post by danbruc » Mon, Jun 27, 2016, 9:22 PM UTC

conjectures wrote: post #61

danbruc wrote: post #40

Earlier quoted context omitted.

But now you are contradicting yourself, aren't you? We agree that for any finite sequence of tosses it is unlikely but possible to get all heads and therefore it is possible to not converge to 0.5. The question is what difference it makes to go from a large but finite to an infinite number of tosses. Either it is impossible to get only heads an infinite number of times, then I have a problem understanding why that is…

When heads has probability p the probability for any n-length sequence of all heads is p^n. For p between 0 and 1 this vanishes as n goes to infinity. The reason is that the conjunction of some events is always less (than or equal to) probable than the events themselves. This should make intuitive sense as the middle bit of a Venn diagram. Even more so for a conjunction of events with a conjunction of events. When ta…

I have no problem with that, all infinite sequences of coin tosses have probability zero. The question is whether a infinite sequence of all heads is an obtainable outcome which would be an counterexample to the statement that the relative frequency converges to 0.5 in the limit.