Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

mcgrayne.com

Post reply on HN
Print view View original

31–40 of 83 posts

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#31

Post by rer0tsaz »

danbruc wrote: post #18

Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair. And just saying that you will obtain the same or at least similar number of heads and tails with high…

No, but let me quote Jaynes (Probability Theory: the Logic of Science, 10.10):

> ‘When I toss a coin, the probability for heads is one-half.’ [...] the issue is between the following two interpretations:

> (A) ‘The available information gives me no reason to expect heads rather than tails, or vice versa – I am completely unable to predict which it will be.’

> (B) ‘If I toss the coin a very large number of times, in the long run heads will occur about half the time – in other words, the frequency of heads will approach 1/2.’

These are not the same except for special circumstances like controlled experiments. Frequentism usually assumes or restricts itself to those special circumstances. The long run here means `for any ε > 0, the probability that the observed frequency n/N lies in the interval (1/2±ε) goes to 1 as N goes to infinity'.

There is no such thing as an intrinsic probability, a coin only has a chance of landing heads when tossed. If we know everything about the coin and how it is tossed we could calculate the result. Some ways of tossing a fair coin are biased. Some coins are biased when tossed in fair ways. A fair coin toss exaggerates factors that are difficult to know and control exactly, like the force that we apply on the coin with our finger.

Jaynes thinks that `true randomness' such as is postulated by conventional quantum physics is unscientific (10.7). In any case it doesn't matter for calculating, and I don't think many Bayesians lose sleep over whether it's unknown, unknowable or `true randomness'.

If the Bayesian has an informative prior, they won't be mislead too much. If a Bayesian is 100% sure beforehand with a δ(x - 0.5) prior they won't be mislead at all, but of course no one is ever 100% sure (Cromwell's rule). On the other hand a frequentist might say p<0.05 and be mislead.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#32

Post by jordigh »

danbruc wrote: post #18

Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair. And just saying that you will obtain the same or at least similar number of heads and tails with high…

> Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means?

Yeah, Kolmogorov's axioms:

https://en.wikipedia.org/wiki/Probability_axioms

To interpret these axioms for 50% probability means that the measure underneath the density function corresponding to the event "heads" is one-half.

But "rigourous" doesn't have anything to do with the natural world. You can't make physics "rigourous", for example, but you can make the mathematics inspired by physics rigourous. Kolmogorov's axioms just give a mathematical description of probability in a formal sense i.e. only discussing its form, not its meaning. Formalism is about saying, "whatever this means, this is how it should behave". It's a very 20th century notion of mathematics.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#33

Post by jordigh »

tgb wrote: post #29
jordigh wrote: post #24

The biggest historical obstacle to Bayesian stats was low amounts of data available and difficulty of computation. Frequentist stats is optimised around these. With 30 samples and very easy computations, you are able to produce a reasonable frequentist confidence interval, sometimes even with less data. On the other hand, even the simplest Bayesian analysis of determining the probability of heads in a coin flip requi…

> On the other hand, even the simplest Bayesian analysis of determining the probability of heads in a coin flip requires some interesting integration and the somewhat obscure Beta distribution. Isn't this kind of misleading? The end result of the Beta distribution, etc. is just the extremely simple-to-compute Rule of Succession [1]. [1] https://en.wikipedia.org/wiki/Rule_of_succession

A bit, but in general, computing posterior distributions tends to very quickly lead to more complicated integrals. It just so happens that the Beta distribution is somewhat nice and symmetrical, if a bit obscure.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#34

Post by danbruc »

daemonomania wrote: post #27
danbruc wrote: post #22

Earlier quoted context omitted.

IANAM either, but that seems to be wrong to me. For any finite number of tosses you may certainly not converge to 0.5 even if it becomes very unlikely pretty quickly not to do so. In order for this to be true, it would have to be literally impossible to always get heads on every toss no matter how often you try. But it seems counter-intuitive at the very least that it is inevitable to get tails eventually. What rules…

If a coin were flipped arbitrarily many times, and it landed on heads each time, and if we had no other information about the coin , a frequentist would say that the probability the coin lands on heads is one. Now, you might protest and say, "But this is a fair coin that just happened to land on heads arbitrarily many times". But then you are relying on a prior definition of probability in order to justify your objec…

I am not sure if that really addresses my issue. My problem is that I don't see what eventually forces convergences to 50/50. We can use two or even better many coins or, at least superficially equivalent, one coin and let several experimenters take turns. The results will converge towards 50/50 but only with high probability, or at least I don't see why they alway would, i.e. why not at least one sequence could not converge indefinitely. And that seems to lead to an infinite tower of experiments.

When tossing a coin infinitely often the ratio of heads and tails will converge to 50/50 but only in that sense that you have to toss the coin an infinite number of times an infinite number of times. And you can't really stop here because you may still deviate arbitrarily far from the expected outcome. So you toss the coin an infinite number of times an infinite number of times an infinite number of times. And you have to keep nesting your experiments until you made an infinite tower of experiments.

And I have no idea if that would be sufficient to ensure that you can no longer deviate from the expected outcome, my intuition tends towards no, but then again this is certainly far outside of the realm of things for which I would intuition expect to work. Likely I am really just confused and missing something obvious.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#35

Post by kareemsabri »

danbruc wrote: post #18

Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair. And just saying that you will obtain the same or at least similar number of heads and tails with high…

> Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair.

It doesn't really require a theory, outside of probability theory itself.

According to the law of large numbers, P(x) = 0.5 does imply that, for N trials, the probability of N/2 outcomes being x (heads) approaches 1 as N approaches infinity. And as you pointed out, the probability of all heads is vanishing, as N approaches infinity. P(all heads) = 1 - P(not all heads), and P(all heads) approaches 0 as N approaches infinity, so P(not all heads) must approach 1. The same exercise could be done with the relative distributions of heads and tails, in a large number of trials.

Each coin flip is random, but the results will converge to the mean. You will obtain the same number of heads and tails with certainty, not high probability, if you can make N big enough.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#36

Post by tingol »

amasad wrote: post #7
arcanus wrote: post #2

Bayesian here. 'The theory that would not die' is a wonderful read, and notes how many scientists used Bayesian techniques (subjective probability) in a variety of contexts while the field was still unpopular (arguably, heretical) in the mainstream statistical community. Bayes was nevertheless used to inform ballistics calculations, help crack the enigma code, or inform search patterns for lost nuclear weapons. I don…

As someone new to Bayesianism I'd be interested in hearing your experience applying it in day-to-day life. How useful do you think it is to the ordinary Joe? From my brief experience, after learning Bayes, my intuition about things involving probabilities grew very different than the people around me. For example, my friends were planning a skydiving trip and I googled the name of the skydiving business and found tha…

I think your friends were right. Peoples lives are not the same as a coin flip. You could almost surely say that whatever caused the accidents in the past has been a point of focus for the agency specifically so it never happens again. You can't really apply your statistical thinking in that scenario.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#37

Post by paol »

danbruc wrote: post #28
paol wrote: post #25

Earlier quoted context omitted.

> For any finite number of tosses you can certainly not converge to 0.5 I think you mean "may not" rather than "can not". But yes, that's why you need to resort to limits[0] to understand the problem. [0] https://en.wikipedia.org/wiki/Limit_of_a_sequence

Thanks, fixed. It understand that it is the limit, but the question I am asking is whether it is possible to always get heads, an infinite number of times. If this is possible, then it is not true that the relative frequency always approaches 0.5 as the number of repetitions goes to infinity. I understand that the probability of obtaining heads an infinite number of times is zero, but does that mean it is impossible?…

> the question I am asking is whether it is possible to always get heads, an infinite number of times

No. By my definition (lay, may be wrong), the probability is the ratio of heads to tails to which we converge at the limit (infinity). The only way for it to converge to 0 is if the probability of heads was 0 to begin with, which would contradict the initial assertion that it was a fair coin.

By the way, from your previous comment it seems to me that you're struggling with getting an intuitive grasp on arguments that rely on infinity. To most people (me included for sure) it's initially difficult to accept that infinity is qualitatively different than a finite amount. Our brain seems to superficially accept the concept, but really it keeps trying to reason about it as a really large (but still finite) quantity.

Are you familiar with the argument of whether 0.999... equals 1? (Spoiler: it does) Some people have a really hard time coming to terms with it, and the fundamental problem for many comes down to the same difficulty of reasoning about infinity. Also, it's a fun way to troll people :)

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#38

Post by danbruc »

kareemsabri wrote: post #35
danbruc wrote: post #18

Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair. And just saying that you will obtain the same or at least similar number of heads and tails with high…

> Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair. It doesn't really require a theory, outside of probability theory itself. According to the law of l…

That means it is totally impossible to get H H H H... or H T T H T T... ad infinitum? I really have no well-founded opinion on that, it just seems very counterintuitive that the coin is not allowed to yield any such sequence, they same as good as any other sequence to me.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#39

Post by kareemsabri »

danbruc wrote: post #38
kareemsabri wrote: post #35

Earlier quoted context omitted.

> Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair. It doesn't really require a theory, outside of probability theory itself. According to the law of l…

That means it is totally impossible to get H H H H... or H T T H T T... ad infinitum? I really have no well-founded opinion on that, it just seems very counterintuitive that the coin is not allowed to yield any such sequence, they same as good as any other sequence to me.

Not at all! It yields that sequence all the time, in small numbers. But, it is impossible to get only heads, if you could truly flip a coin infinitely many times, which of course you can't. The larger you make the sequence, the smaller you make the probability of all heads. In your example:

P(H,H,H,H) = 0.5 ^ 4 = 0.0625 P(H,H,H,H,H,H,H,H) = 0.5 ^ 8 = 0.00390625

As you can see it's getting pretty small already. However, the probably will never actually reach 0. It's the half step concept, if you keep halving the distance between yourself and an object, you will never actually reach it. But the distance between yourself and the object will approach 0 as the number of steps approaches infinity. 0.5 ^ infinity = 0

Note that P(H|H,H,H) = probability of heads, given 3 previous heads, is still just 0.5. Any independent coin flip is 0.5 chance of being heads. But, when you've done 0 coin flips, and you ask, what are the odds of all heads for the next, say, million flips, they are nearly 0.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#40

Post by danbruc »

kareemsabri wrote: post #39
danbruc wrote: post #38

Earlier quoted context omitted.

That means it is totally impossible to get H H H H... or H T T H T T... ad infinitum? I really have no well-founded opinion on that, it just seems very counterintuitive that the coin is not allowed to yield any such sequence, they same as good as any other sequence to me.

Not at all! It yields that sequence all the time, in small numbers. But, it is impossible to get only heads, if you could truly flip a coin infinitely many times, which of course you can't. The larger you make the sequence, the smaller you make the probability of all heads. In your example: P(H,H,H,H) = 0.5 ^ 4 = 0.0625 P(H,H,H,H,H,H,H,H) = 0.5 ^ 8 = 0.00390625 As you can see it's getting pretty small already. Howeve…

But now you are contradicting yourself, aren't you? We agree that for any finite sequence of tosses it is unlikely but possible to get all heads and therefore it is possible to not converge to 0.5. The question is what difference it makes to go from a large but finite to an infinite number of tosses. Either it is impossible to get only heads an infinite number of times, then I have a problem understanding why that is, or all heads is still a possible outcome even in the infinite case, then the process does not necessarily yield 0.5 even in the limit of an infinite number of tosses.
Post reply on HN