Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beautiful Probability

readthesequences.com

Post reply on HN
Print view View original

41–50 of 67 posts

Re: Beautiful Probability

#41

Post by tech_ken »

usgroup wrote: post #11
jawarner wrote: post #3

Isn't that Edwin T. Jaynes example just p-hacking? If only 1 out of 100 experiments produces a statistically significant result, and you only report the one, I would intuitively consider that evidence to be worth less. Can someone more versed in Bayesian statistics better explain the example?

Well no because it’s talking about either a fixed sample size or stopping when a % total is reached. Neither imply a favourable p-value necessarily. I think the author means to say that it’s two methods incidentally equivalent in the data they collect that may draw different conclusions based on their initial assumptions. Question is how do you make coherent sense of it. At level 1 depth it’s insightful. At level 2 d…

> The other ... decided he would not stop until he had data indicating a rate of cures definitely greater than 60%

I believe that "definitely greater than 60%" is supposed to imply that the researcher is stopping when the p-value of their HA (theta>=60%) is below alpha, so an optional stopping (ie. "p-hacking") situation.

Re: Beautiful Probability

#42

Post by pdonis »

AbrahamParangi wrote: post #18

I'm confused in that I don't see how this is troubling. Yes, the two experimenters rolled dice and got the same result, but it's as if one of them was rolling a 6 sided die and the other a 20 sided one. Each experiment is not a result per se but a sample from a distribution. How you infer the shape of that distribution based on the experiment is a function of the distribution of all courses your experiment could have…

> Each experiment is not a result per se but a sample from a distribution.

But what distribution? What is this "distribution" that we are taking a sample from?

The frequentist says: because the two experimenters have different intentions, the experiments they ran are samples from different distributions.

But the Bayesian says: the experimenter's intentions can't affect things like how dice rolls come out or how well a given treatment works on a given patient. The actual "distribution" is the set of all factors that do affect how the dice rolls come out or how well the treatment works on each patient. And those factors are the same for both experimenters; their different intentions don't affect that. So both sets of data are samples from the same distribution, not different ones.

> How you infer the shape of that distribution based on the experiment is a function of the distribution of all courses your experiment could have taken.

If you're going to state it this way, then the Bayesian response is: "all courses your experiment could have taken" has nothing to do with the experimenter's intentions. The experimenters can't magically make the physical world and the biology of humans work differently depending on what stopping criterion they choose. And the physical world and the biology of humans is what determines "the courses your experiment could have taken".

In other words, when the frequentist makes up "distributions" based on the experimenter's stopping criterion, they are, whether they admit it (or even realize it) or not, making a claim about how the physical world and the biology of humans works that is obviously false.

Re: Beautiful Probability

#43

Post by pdonis »

ninthcat wrote: post #40
lalaithion wrote: post #30

Earlier quoted context omitted.

Both events have the same probability of happening; 1/20. The fact that the researcher intended to do something in a reality that didn't happen isn't relevabnt.

If you want to know whether a drug is more effective than placebo, the answer to that question depends on both the data collected in a study and the initial study design. There’s a reason why it’s meaningless to say “that was unlikely” after somebody says they were born on January 1, or after getting a two-factor code that is the same number six times. There’s nothing special about those particular events except for…

> a flawed experimental design where you repeat the experiment until you get the results you want to see.

But the Bayesian point is that, if you use Bayesian statistics, this doesn't work. Except by outright lying about their experimental protocol or the data that was actually collected (for example, only reporting the successful trial at the end and not all the failed ones the preceded it), an experimenter cannot "fool" you into accepting a hypothesis not justified by the data. They can point to the one successful trial all they want, and make up stories about how the previous failed trials were somehow different, and the Bayesian simply does not care. The Bayesian just looks at the entire corpus of data and finds that it doesn't support the hypothesis, and that's it.

Re: Beautiful Probability

#44

Post by pdonis »

d0mine wrote: post #4

Bayesian approach sounds like a religion (one true way). There is nothing unusual about different mathematical methods/models producing different results e.g., the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest (sometimes they do/sometimes they don't). All models are wrong some are useful.

> Bayesian approach sounds like a religion (one true way).

Only about the things that can be mathematically proven. Which is just like any other branch of math.

It is true that some Bayesians (and EY can be argued to be among them) like to talk as though Bayesian computation is a drop-in replacement for your brain. Of course it isn't, and Bayesianism, like any mathematical approach, should be taken with a good-sized dose of humility. As Bertrand Russell said, to the extent that mathematical propositions refer to reality, they are not certain, and to the extent that they are certain, they do not refer to reality.

Re: Beautiful Probability

#45

Post by pdonis »

d0mine wrote: post #4

Bayesian approach sounds like a religion (one true way). There is nothing unusual about different mathematical methods/models producing different results e.g., the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest (sometimes they do/sometimes they don't). All models are wrong some are useful.

> the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest

No. The number of roots that you care about might depend on your private thoughts; but the number of roots itself does not. It's a mathematical fact. It just might not be a mathematical fact that you actually care about. But what you care about is not part of math.

Re: Beautiful Probability

#46

Post by AbrahamParangi »

pdonis wrote: post #42
AbrahamParangi wrote: post #18

I'm confused in that I don't see how this is troubling. Yes, the two experimenters rolled dice and got the same result, but it's as if one of them was rolling a 6 sided die and the other a 20 sided one. Each experiment is not a result per se but a sample from a distribution. How you infer the shape of that distribution based on the experiment is a function of the distribution of all courses your experiment could have…

> Each experiment is not a result per se but a sample from a distribution. But what distribution? What is this "distribution" that we are taking a sample from? The frequentist says: because the two experimenters have different intentions , the experiments they ran are samples from different distributions. But the Bayesian says: the experimenter's intentions can't affect things like how dice rolls come out or how well…

This seems to assume that intentions "don't count" in some way, as if they were nonphysical, whereas unless you presume a supernatural soul, they are as physical as any other part of the experiment.

Re: Beautiful Probability

#47

Post by [deleted] »

jawarner wrote: post #3

Isn't that Edwin T. Jaynes example just p-hacking? If only 1 out of 100 experiments produces a statistically significant result, and you only report the one, I would intuitively consider that evidence to be worth less. Can someone more versed in Bayesian statistics better explain the example?

[deleted]

Re: Beautiful Probability

#48

Post by lalaithion »

AbrahamParangi wrote: post #25
lalaithion wrote: post #21

Earlier quoted context omitted.

If you see two people roll a d20 and get a 20, you get to say "wow, that was unlikely" to both of them, even if one of them privately admits they were going to quickly re-roll their die if they got below a 10. What matters is their actual behavior (identical in the example) not their intentions. The d6 vs d20 version is different because their behavior is different.

Let's imagine that we ran it as a simulation and we ran it a million times. The two people would have a different distribution of results. If you ignore the intention, you ignore reality as if that intention were not a part of it. Do you not notice that your inference is less accurate using this line of reasoning? Does that not suggest that it's simply wrong?

Here’s another example:

Say you have a lazy researcher. They flip a coin, and if it comes up heads, they do the experiment. If it comes up tails, they just write down a random number.

If you _only get access_ to the final number, then you should discount what they wrote down – it’s 50% likely to be fake.

If you do 1,000,000 simulations of this, it’s useless 50% of the time.

But if you know the result of the coin flip, it doesn’t matter whether they would have generated a nonsense number in a different timeline, or that they’re not reliably accurate. _You know_ they’re reliably accurate in _this case_, so you can trust their data.

Re: Beautiful Probability

#49

Post by pdonis »

AbrahamParangi wrote: post #46
pdonis wrote: post #42

Earlier quoted context omitted.

> Each experiment is not a result per se but a sample from a distribution. But what distribution? What is this "distribution" that we are taking a sample from? The frequentist says: because the two experimenters have different intentions , the experiments they ran are samples from different distributions. But the Bayesian says: the experimenter's intentions can't affect things like how dice rolls come out or how well…

This seems to assume that intentions "don't count" in some way, as if they were nonphysical, whereas unless you presume a supernatural soul, they are as physical as any other part of the experiment.

> This seems to assume that intentions "don't count" in some way, as if they were nonphysical

Not nonphysical: just not part of the physical degrees of freedom that can affect things like how die rolls come up or how well a given treatment works on a patient.

The experimenter's intentions (not about the stopping criterion, but about other things) can of course be upstream physical causes, so to speak, of things like what the actual process of the treatment is, and that can, of course affect how well the treatment works. But in the scenario under discussion, all those things are stipulated to be the same in both experiments. And once that is specified, whatever physical variation corresponds to the variation in the experimenters' intentions cannot affect the results.

Re: Beautiful Probability

#50

Post by drdeca »

usgroup wrote: post #26
lalaithion wrote: post #16

Earlier quoted context omitted.

P(B|I saw E, P) = P(I saw E|B,P) * P(B|P) / P(I saw E|P) P(B|E was false, I saw E, P) = P(E was false|B,I saw E,P) * P(B|P,I saw E) / P(E was false|P, I saw E) This is a pretty basic application of Bayes' theorem.

Love it: p(I saw E) and p(I didn’t really see E). Just move the argument one level down: “I saw E is false” and it turns out so is “E is false” . So then? Add “E was false was false”? Turtles all the way down. At some point something has to be “true” in order to conditionalise on it.

I believe you can condition on a probability of proposition.

For example, if you are in a fairly dark room and you observe with 90% confidence a red object. Then you can do (iirc) P(X | 90% confidence see red object) = 90% * P(X | see red object) + 10% * P(X | do not see red object)

I would think that in principle, this allows for allowing all observations to be fallible, without any kind of “infinite regress” problem? You just apply the same kind of process each time.

Post reply on HN