Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beautiful Probability

readthesequences.com

Post reply on HN
Print view View original

11–20 of 67 posts

Re: Beautiful Probability

#11

Post by usgroup »

jawarner wrote: post #3

Isn't that Edwin T. Jaynes example just p-hacking? If only 1 out of 100 experiments produces a statistically significant result, and you only report the one, I would intuitively consider that evidence to be worth less. Can someone more versed in Bayesian statistics better explain the example?

Well no because it’s talking about either a fixed sample size or stopping when a % total is reached. Neither imply a favourable p-value necessarily.

I think the author means to say that it’s two methods incidentally equivalent in the data they collect that may draw different conclusions based on their initial assumptions. Question is how do you make coherent sense of it.

At level 1 depth it’s insightful.

At level 2 depth it’s a straw man.

At level 3 depth, just keep drinking until you’re back at level 1 depth.

Re: Beautiful Probability

#12

Post by biomcgary »

jawarner wrote: post #8
usgroup wrote: post #5

So you know when you believe something and then you update your belief because you get some evidence? Yeah, and then you stack some beliefs on top of that. And then you discover the evidence wasn’t actually true. Remind me again what the normative Bayesian update looks like in that instance. Unfortunately it’s turtles all the way down.

Real world systems are complicated. In theory, you could do belief propagation to update your beliefs through the whole network, if your brain worked something like a Bayesian network.

Natural selection didn't wire our brains to work like a Bayesian network. If it had, wouldn't it be easier to make converts to the Church of Reverend Bayes? /s

Alternatively, brains ARE Bayesian networks with hard coded priors that cannot be changed without CRISPR.

Re: Beautiful Probability

#13

Post by usgroup »

d0mine wrote: post #4

Bayesian approach sounds like a religion (one true way). There is nothing unusual about different mathematical methods/models producing different results e.g., the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest (sometimes they do/sometimes they don't). All models are wrong some are useful.

Yeah I’d agree at some depth. We don’t talk enough about integers, rationals and real numbers and what they imply for our “normative rationality” or “epistemological commitment”. But aside from the integers, everything else is totally suspicious.

Re: Beautiful Probability

#14

Post by biomcgary »

d0mine wrote: post #4

Bayesian approach sounds like a religion (one true way). There is nothing unusual about different mathematical methods/models producing different results e.g., the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest (sometimes they do/sometimes they don't). All models are wrong some are useful.

One of my priors: "a group of people who look like a faith-based community, but claim not to be one, should not be trusted".

Re: Beautiful Probability

#15

Post by Terr_ »

jawarner wrote: post #3

Isn't that Edwin T. Jaynes example just p-hacking? If only 1 out of 100 experiments produces a statistically significant result, and you only report the one, I would intuitively consider that evidence to be worth less. Can someone more versed in Bayesian statistics better explain the example?

I think the point is that the different planned stopping rules of each researcher--their subjective thoughts--should not affect what we consider the objective or mathematical significance of their otherwise-identical process and results. (Not unless humans have psychic powers.)

It's illogical to deride one of those two result-sets as telling us less about the objective universe just because the researcher had a different private intent (e.g. "p-hacking") for stopping at n=100.

_________________

> According to old-fashioned statistical procedure [...] It’s quite possible that the first experiment will be “statistically significant,” the second not. [...]

> But the likelihood of a given state of Nature producing the data we have seen, has nothing to do with the researcher’s private intentions. So whatever our hypotheses about Nature, the likelihood ratio is the same, and the evidential impact is the same, and the posterior belief should be the same, between the two experiments. At least one of the two Old Style methods must discard relevant information—or simply do the wrong calculation—for the two methods to arrive at different answers.

Re: Beautiful Probability

#16

Post by lalaithion »

usgroup wrote: post #5

So you know when you believe something and then you update your belief because you get some evidence? Yeah, and then you stack some beliefs on top of that. And then you discover the evidence wasn’t actually true. Remind me again what the normative Bayesian update looks like in that instance. Unfortunately it’s turtles all the way down.

    P(B|I saw E, P) = P(I saw E|B,P) * P(B|P) / P(I saw E|P)

    P(B|E was false, I saw E, P) = P(E was false|B,I saw E,P) * P(B|P,I saw E) / P(E was false|P, I saw E)
This is a pretty basic application of Bayes' theorem.

Re: Beautiful Probability

#17

Post by lalaithion »

d0mine wrote: post #4

Bayesian approach sounds like a religion (one true way). There is nothing unusual about different mathematical methods/models producing different results e.g., the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest (sometimes they do/sometimes they don't). All models are wrong some are useful.

> the number of roots even for the same quadratic equation may depend on "private" thoughts such as whether complex roots are of interest

You are confusing ambiguity in a problem statement due to human language being imprecise with two well-specified identical experimental results having different results due to the intentions of the human carrying them out.

Is arithmetic a religion because there's "one true way" of adding integers?

Re: Beautiful Probability

#18

Post by AbrahamParangi »

I'm confused in that I don't see how this is troubling. Yes, the two experimenters rolled dice and got the same result, but it's as if one of them was rolling a 6 sided die and the other a 20 sided one. Each experiment is not a result per se but a sample from a distribution.

How you infer the shape of that distribution based on the experiment is a function of the distribution of all courses your experiment could have taken. This set of paths is different in each case, which means the inference we make must also be different.

There is no inconsistency. The confusion seems to be in assuming that the experimental result was a true statement about the nature of the world rather than a true statement about simply what happened.

edit: This seems to me to be a specific case of a general class of difficult thinking where you ask yourself: "what are all the worlds that I might be in that are consistent with what I'm presently observing".

Re: Beautiful Probability

#19

Post by abeppu »

skulk wrote: post #6
jawarner wrote: post #3

Isn't that Edwin T. Jaynes example just p-hacking? If only 1 out of 100 experiments produces a statistically significant result, and you only report the one, I would intuitively consider that evidence to be worth less. Can someone more versed in Bayesian statistics better explain the example?

I find the original discussion to be far more interesting than whatever I just read in TFA: https://books.google.com.mx/books?id=sLz0CAAAQBAJ&pg=PA13&lp...

> One who thinks that the important question is: "Which quantities are random?" is then in this situation. For the first researcher, n was a fixed constant, r was a random variable with a certain sampling distribution. For the second researcher, r/n was a fixed constant (approximately), and n was the random variable, with a very different sampling distribution. Orthodox practice will then analyze the two experiments in different ways, and will in general draw different conclusions about the efficacy of the treatment from them.

But so then the data _are_ different between the two experiments, because they were observing different random variables -- so why is it concerning if they arrive at different conclusions? In fact, the _fact that the 2nd experiment finished_ is also an observation on its own (e.g. if the treatment was in fact a dangerous poison, perhaps it would have been infeasible for the 2nd researcher to reach their stopping criteria).

Re: Beautiful Probability

#20

Post by lalaithion »

jawarner wrote: post #3

Isn't that Edwin T. Jaynes example just p-hacking? If only 1 out of 100 experiments produces a statistically significant result, and you only report the one, I would intuitively consider that evidence to be worth less. Can someone more versed in Bayesian statistics better explain the example?

If you have two researchers, and one is "trying" to p-hack by repeating an experiment with different parameters, and one is trying to avoid p-hacking by preregistering their parameters, you might expect the paper published by the latter one to be more reliable.

However, if you know that the first researcher just happened to get a positive result on their first try (and therefore didn't actually have to modify parameters), Bayesian math says that their intentions didn't matter, only their result. If, however, they did 100 experiments and chose the best one, then their intentions... still don't matter! but their behavior does matter, and so we can discount their paper.

Now, if you _only_ know their intentions but not their final behavior (because they didn't say how many experiments they did before publishing), then their intentions matter because we can predict their behavior based on their intentions. But once you know their behavior (how many experiments they attempted), you no longer care about their intentions; the data speaks for itself.

Post reply on HN