How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy • Hacker News Board

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#21

Post by Jommi » Mon, Jun 27, 2016, 11:06 AM UTC

amasad wrote: post #7

arcanus wrote: post #2

Bayesian here. 'The theory that would not die' is a wonderful read, and notes how many scientists used Bayesian techniques (subjective probability) in a variety of contexts while the field was still unpopular (arguably, heretical) in the mainstream statistical community. Bayes was nevertheless used to inform ballistics calculations, help crack the enigma code, or inform search patterns for lost nuclear weapons. I don…

As someone new to Bayesianism I'd be interested in hearing your experience applying it in day-to-day life. How useful do you think it is to the ordinary Joe? From my brief experience, after learning Bayes, my intuition about things involving probabilities grew very different than the people around me. For example, my friends were planning a skydiving trip and I googled the name of the skydiving business and found tha…

Weren't your friends right? Since the incident happened, it's quite likely they fired an incompetnet instructor or tightened security protocols to such an effect that future dives would be safer.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#22

Post by danbruc » Mon, Jun 27, 2016, 11:15 AM UTC

paol wrote: post #20

danbruc wrote: post #18

Is there any (uncontroversial) theory that rigorously defines what a 50 % probability for heads and tails means? It certainly doesn't mean that in the long run you will obtain the same number of heads and tails because there is a (vanishing) chance that you will always get heads even though the coin is actually fair. And just saying that you will obtain the same or at least similar number of heads and tails with high…

> doesn't mean that in the long run you will obtain the same number of heads and tails IANAM, but in my understanding that's exactly what it means, if you express "in the long run" as "the ratio tends to 0.5 as the number of tosses tends toward ∞"

IANAM either, but that seems to be wrong to me. For any finite number of tosses you may certainly not converge to 0.5 even if it becomes very unlikely pretty quickly not to do so. In order for this to be true, it would have to be literally impossible to always get heads on every toss no matter how often you try. But it seems counter-intuitive at the very least that it is inevitable to get tails eventually. What rules out the possibility to get heads over and over again, an infinite number of times? [1] But if it is possible to obtain an infinite number of heads and not a single tails, then it is not true that ratio always converges towards 0.5. In consequence one would have to modify the definition and say that it is only very likely to converge towards 0.5 but that obviously raises the problem that »very likely« sounds a lot like »with high probability« which is the thing we are trying to define, hence this would become a somewhat circular definition.

[1] Actually it must not only be impossible to only ever get heads or tails, it must be impossible to obtain any sequence of outcomes that has not exactly the same number of heads and tails. There is only one sequence of outcomes with only heads and only one with only tails but there is already an infinite number of sequences with all heads but one tails or vice versa. And there is an enormous number of possible sequences with 40 % heads and 60 % tails or any other ratio other than 50/50.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#23

Post by hacker42 » Mon, Jun 27, 2016, 11:18 AM UTC

mikekchar wrote: post #11

amasad wrote: post #7

Earlier quoted context omitted.

As someone new to Bayesianism I'd be interested in hearing your experience applying it in day-to-day life. How useful do you think it is to the ordinary Joe? From my brief experience, after learning Bayes, my intuition about things involving probabilities grew very different than the people around me. For example, my friends were planning a skydiving trip and I googled the name of the skydiving business and found tha…

The one place I've used Bayes (hopefully properly!) is in a spaced repetition flash card program. Usually spaced repetition algorithms wait a certain amount of time based on how many times you have seen and remembered a card. The more times you have remembered it, the longer you wait. It then creates a schedule for each day. You review the cards that have "expired" their wait time. I wanted to turn this upside down.…

That basically seems to ignore the exponential Ebbinghausian forgetting curve. I doubt it results in better scheduling of the cards.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#24

Post by jordigh » Mon, Jun 27, 2016, 11:19 AM UTC

The biggest historical obstacle to Bayesian stats was low amounts of data available and difficulty of computation. Frequentist stats is optimised around these. With 30 samples and very easy computations, you are able to produce a reasonable frequentist confidence interval, sometimes even with less data. On the other hand, even the simplest Bayesian analysis of determining the probability of heads in a coin flip requires some interesting integration and the somewhat obscure Beta distribution.

Now that we have lots of data and lots of computing power, Bayesian stats can show its results, after getting rebranded as "machine learning".

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#25

Post by paol » Mon, Jun 27, 2016, 11:23 AM UTC

danbruc wrote: post #22

paol wrote: post #20

Earlier quoted context omitted.

> doesn't mean that in the long run you will obtain the same number of heads and tails IANAM, but in my understanding that's exactly what it means, if you express "in the long run" as "the ratio tends to 0.5 as the number of tosses tends toward ∞"

IANAM either, but that seems to be wrong to me. For any finite number of tosses you may certainly not converge to 0.5 even if it becomes very unlikely pretty quickly not to do so. In order for this to be true, it would have to be literally impossible to always get heads on every toss no matter how often you try. But it seems counter-intuitive at the very least that it is inevitable to get tails eventually. What rules…

> For any finite number of tosses you can certainly not converge to 0.5

I think you mean "may not" rather than "can not". But yes, that's why you need to resort to limits[0] to understand the problem.

[0] https://en.wikipedia.org/wiki/Limit_of_a_sequence

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#26

Post by twanvl » Mon, Jun 27, 2016, 11:29 AM UTC

hacker42 wrote: post #19

I could never quite understand the divide between Bayesian statistics and frequentist statistics. Both seem to be ultimately about counting the frequency by which something occurs and normalizing this frequency with respect to the number of all possible outcomes. Bayesian statistics essentially is concerned with the application of the Bayesian updating technique by which one can iteratively improve a distribution ove…

Bayesian and frequentist approaches ultimately have a different notion of probability.

In the frequentist approach, a probability of 10% means that if you repeat an experiment many times, roughly 1 out of 10 times you will observe an event.

In Baysian statistics, a probability of 10% means that you are that certain about the event happening. So you would be willing to bet at 10 to 1 odds on the event happening. There doesn't have to be any repetition of the experiment for the probability to make sense. And as you can hopefully see, there is always a prior, that is, your belief about the event before doing any experiments.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#27

Post by daemonomania » Mon, Jun 27, 2016, 11:33 AM UTC

danbruc wrote: post #22

paol wrote: post #20

Earlier quoted context omitted.

> doesn't mean that in the long run you will obtain the same number of heads and tails IANAM, but in my understanding that's exactly what it means, if you express "in the long run" as "the ratio tends to 0.5 as the number of tosses tends toward ∞"

IANAM either, but that seems to be wrong to me. For any finite number of tosses you may certainly not converge to 0.5 even if it becomes very unlikely pretty quickly not to do so. In order for this to be true, it would have to be literally impossible to always get heads on every toss no matter how often you try. But it seems counter-intuitive at the very least that it is inevitable to get tails eventually. What rules…

If a coin were flipped arbitrarily many times, and it landed on heads each time, and if we had no other information about the coin, a frequentist would say that the probability the coin lands on heads is one.

Now, you might protest and say, "But this is a fair coin that just happened to land on heads arbitrarily many times". But then you are relying on a prior definition of probability in order to justify your objection to the frequentist's claim, since presumably what you mean by "fair" is that the probability the coin lands on heads is 1/2.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#28

Post by danbruc » Mon, Jun 27, 2016, 11:37 AM UTC

paol wrote: post #25

danbruc wrote: post #22

Earlier quoted context omitted.

IANAM either, but that seems to be wrong to me. For any finite number of tosses you may certainly not converge to 0.5 even if it becomes very unlikely pretty quickly not to do so. In order for this to be true, it would have to be literally impossible to always get heads on every toss no matter how often you try. But it seems counter-intuitive at the very least that it is inevitable to get tails eventually. What rules…

> For any finite number of tosses you can certainly not converge to 0.5 I think you mean "may not" rather than "can not". But yes, that's why you need to resort to limits[0] to understand the problem. [0] https://en.wikipedia.org/wiki/Limit_of_a_sequence

Thanks, fixed. It understand that it is the limit, but the question I am asking is whether it is possible to always get heads, an infinite number of times. If this is possible, then it is not true that the relative frequency always approaches 0.5 as the number of repetitions goes to infinity. I understand that the probability of obtaining heads an infinite number of times is zero, but does that mean it is impossible? After all any specific infinite sequence of outcomes has probability zero.

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#29

Post by tgb » Mon, Jun 27, 2016, 11:43 AM UTC

jordigh wrote: post #24

The biggest historical obstacle to Bayesian stats was low amounts of data available and difficulty of computation. Frequentist stats is optimised around these. With 30 samples and very easy computations, you are able to produce a reasonable frequentist confidence interval, sometimes even with less data. On the other hand, even the simplest Bayesian analysis of determining the probability of heads in a coin flip requi…

> On the other hand, even the simplest Bayesian analysis of determining the probability of heads in a coin flip requires some interesting integration and the somewhat obscure Beta distribution.

Isn't this kind of misleading? The end result of the Beta distribution, etc. is just the extremely simple-to-compute Rule of Succession [1].

[1] https://en.wikipedia.org/wiki/Rule_of_succession

Re: How Bayes’ Rule Emerged Triumphant from Two Centuries of Controversy

#30

Post by jessriedel » Mon, Jun 27, 2016, 11:44 AM UTC

asher_ wrote: post #9

jessriedel wrote: post #8

Earlier quoted context omitted.

The particle physics do sometimes make Bayesian plots and end up arguing over priors. See this, for example: https://cds.cern.ch/record/1375842/files/ATL-PHYS-PUB-2011-0... And I've heard more than one confused argument between particle physicists about whether Bayesianism or frequentism is "right"...

Given the high level of statistical certainty that particle physicists strive for from experimental data, how much of a difference does the approach matter by the point where they are confident enough to endorse the theory? For instance, if you are going for five sigma (like in the Higgs Boson discovery), how would the different techniques change when the data is considered sufficient, if at all?

The five-sigma threshold is a limit for making big public announcements, or claiming definitive discrete discoveries (like new particles) that non-particle physicists can take for granted. But there's lots of intermediate work done at accelerators that can't reach that threshold but still informs further work, and even influences the construction and design of the next multi-billion dollar machines. The bi-annual Review of Particle Physics is as thick as a phone book and filled with (usually) 95%-confidence intervals for measured values. For instance, here is the 89-page summary of what is known about the tau particle:

http://pdg.lbl.gov/2015/listings/rpp2015-list-tau.pdf