The problem with p-values • Hacker News Board

Re: The problem with p-values

#11

Post by jsprogrammer » Tue, Oct 11, 2016, 5:46 PM UTC

>The problem is that the p-value gives the right answer to the wrong question. What we really want to know is not the probability of the observations given a hypothesis about the existence of a real effect, but rather the probability that there is a real effect – that the hypothesis is true – given the observations. And that is a problem of induction.

The problem of induction is real and unavoidable in the general case, but there is no such thing as "the probability that there is a real effect". Either there is a "real effect" or there is not.

It might be possible to find a "probability" that you observed a "real effect".

>The problem of induction was solved, in principle, by the Reverend Thomas Bayes in the middle of the 18th century.

The problem of induction is fundamentally unsolvable (hence, "problem"). The article just states that it was solved and never mentions it again. Is it a widely held view that induction was solved by Bayes? Does anyone know where I can read more detailed claims about how people believe Bayes solved induction?

Re: The problem with p-values

#12

Post by nkurz » Tue, Oct 11, 2016, 7:37 PM UTC

bbctol wrote: post #7

nkurz wrote: post #3

Take the proposition that the Earth goes round the Sun. It either does or it doesn’t, so it’s hard to see how we could pick a probability for this statement. Sometimes I fail to follow the distinctions made in certain strains of classical statistics. How is his "conundrum" different than this one? "Roll a 6 sided die whose result we cannot see. Take the proposition that the top shows a 6. It either does or it doesn’t…

Well, it's less of a conundrum in the case of the die because it's easier to estimate the priors. That's the power and difficulty of using Bayesian reasoning; you need some estimate of how likely something is to be true before you perform the statistical test. In case of the six-sided die, we have a good physical model and years of memory to know that there should be a 1/6 chance of any number coming up, so it's easy…

Well, it's less of a conundrum in the case of the die because it's easier to estimate the priors.

I agree, but in the case of a die, "estimating the priors" is the same thing as "knowing the probability of a 6". So you are saying (correctly) that once we have a model that we believe in, we can assign a probability because we believe we know the probability in advance.

I was pointing more to the author's emphasis on "either it does or it doesn't". One view is that probability requires replication, and we can only speak of probabilities in the long run of many trials. The other view (Bayesian to my limited understanding) is that probability can also be used to measure appropriate degree of belief in a proposition.

While it's hard to quantify, and dependent on one's assumptions, I do think it's possible to speak of "the probability that the earth revolves around the sun", much as I think it's meaningful to talk about "the probability of anthropogenic warming" even though we only have a single Earth to study.

I don't really understand how classical statistics disallows discussion of the first while allowing discussion of the second. Maybe it doesn't?

Re: The problem with p-values

#13

Post by bbctol » Tue, Oct 11, 2016, 8:31 PM UTC

nkurz wrote: post #12

bbctol wrote: post #7

Earlier quoted context omitted.

Well, it's less of a conundrum in the case of the die because it's easier to estimate the priors. That's the power and difficulty of using Bayesian reasoning; you need some estimate of how likely something is to be true before you perform the statistical test. In case of the six-sided die, we have a good physical model and years of memory to know that there should be a 1/6 chance of any number coming up, so it's easy…

Well, it's less of a conundrum in the case of the die because it's easier to estimate the priors. I agree, but in the case of a die, "estimating the priors" is the same thing as "knowing the probability of a 6". So you are saying (correctly) that once we have a model that we believe in, we can assign a probability because we believe we know the probability in advance. I was pointing more to the author's emphasis on "…

I'd say classical statistics lets you deduce, based on some knowledge that something is the case, the probability of an event. It doesn't let you induce the probability of something being the case (that's why we have all this indirect dodging with p-values.) So I think there is a fundamental difference between "What are the odds of rolling a six" as "What are the odds of that event happening," and "What are the odds the Sun revolves around the Earth" as a question about whether or not something is true.

That's what the author seems to mean by "either it does or it doesn't," though it isn't worded that well. Even if the die has already been rolled, we know that it could have been a 4 or a 5, and have good knowledge of the probabilities of those events. This is more like looking at a picture of the top of a die and seeing a six: what are the odds that it's a standard die, vs. a die with sixes on each face? I don't think it's meaningful to talk about that, or the sun going around the Earth, without using a Bayesian prior.

Re: The problem with p-values

#14

Post by bayeslives » Tue, Oct 11, 2016, 11:29 PM UTC

jsprogrammer wrote: post #11

>The problem is that the p-value gives the right answer to the wrong question. What we really want to know is not the probability of the observations given a hypothesis about the existence of a real effect, but rather the probability that there is a real effect – that the hypothesis is true – given the observations. And that is a problem of induction. The problem of induction is real and unavoidable in the general ca…

Bayes did not "solve induction". If we define induction as telling which specific model generated this data: that is not possible. Countless models could in theory generate our data. What we need is some restriction. Like a prior. And when was the last time you started a research project without any idea what to expect? And if you did, wouldn't it be wiser to do some literature study, experts interviews etc. before starting experiments? Modeling the state of the art, pre-experiment, seems like a clever move anyway.

To name just a few of the NHST/p-value flaws:

2- it is vulnerable to wrong interpretations. * No, a 95% confidence interval (a,b) does NOT mean there is a 95% chance that Mu is in (a,b). * No, p=0.04 does not mean they is a 96% chance that H1 is true.

3- the p-value depends on the intentions of the scientist. If you end your experiment after 80 observations, as planned, your p-value is different from that of an experiment that ended unplanned after 80 observations. So the same data have different evidential power, influenced by results you did not see in experiments you did not do. This is very unsatisfactory.

4- the idea of "an effect that exists or does not exist", based on some arbitrary threshold. The reality is, in many cases, uncertainty and variation. In group A I see effects of medicine A, with lots of variation between persons. In group B I see varying effects of medicine B. Then I introduce uncertainty by drawing random samples from A and B. Let's day I used those samples to make an inference: is, on average, medicine A better than medicine B ? Matras like "there is an effect, or there isn't" are not very helpful. Statistics should be about quantifying uncertainty rather than give false yes/no statements.

5. Basing decisions and knowledge on the data only makes t vulnerable to outliers, unlucky samples and so on. And why should you NOT use information, when it's there ?

Re: The problem with p-values

#15

Post by qwrusz » Wed, Oct 12, 2016, 1:04 AM UTC

It's ironic the author starts off saying they need to get more rigorous in their science and statistics. Then goes on to write a relatively short and not very rigorous overview of Bayesian statistics.

Statistics and probability are difficult subjects. They are also not intuitive subjects for many people.

Besides the publish or perish thing, I would guess many authors of these unreliable/non-replicable biomedicine papers were focused on biomedicine during grad school - while they learned statistics it was a secondary subject.

A solution seems to be more statistics training for academics doing studies or requiring a trained statistician on the team/reviewing a study prior to publication. I don't see either happening to be honest, and this problem will likely continue.

Re: The problem with p-values

#16

Post by duneroadrunner » Wed, Oct 12, 2016, 4:10 AM UTC

bbctol wrote: post #13

nkurz wrote: post #12

Earlier quoted context omitted.

Well, it's less of a conundrum in the case of the die because it's easier to estimate the priors. I agree, but in the case of a die, "estimating the priors" is the same thing as "knowing the probability of a 6". So you are saying (correctly) that once we have a model that we believe in, we can assign a probability because we believe we know the probability in advance. I was pointing more to the author's emphasis on "…

I'd say classical statistics lets you deduce, based on some knowledge that something is the case, the probability of an event. It doesn't let you induce the probability of something being the case (that's why we have all this indirect dodging with p-values.) So I think there is a fundamental difference between "What are the odds of rolling a six" as "What are the odds of that event happening," and "What are the odds…

> So I think there is a fundamental difference between "What are the odds of rolling a six" as "What are the odds of that event happening," and "What are the odds the Sun revolves around the Earth" as a question about whether or not something is true.

Right, using Bayes makes the difference not a "fundamental" one, but just a practical one of coming up with the Bayesian prior. Even if it would be hard to establish a consensus on the most appropriate complete set of factors determining the Bayesian prior, there are clearly some examples of meaningful inputs. Like, for example, if you somehow had information about how many of the researcher's previous hypotheses on the subject failed to obtain a "p-value of significance".

But perhaps more practically, you could consider things like the (Kolmogorov) complexity of the hypothesis. Since the number of "low complexity" hypotheses are finite, they are less susceptible to "p-value mining"[1]. The challenge being deciding which inputs to use to evaluate the complexity.

This seems to me like an area where machine learning should be applicable. Rather than lament about the impracticality of determining an appropriate, tractable set of (quantifiable) criteria for determining a Bayesian prior, why not just include every potentially relevant piece of information and let Deep Thought[2] figure out which are actually relevant? So what we really need is unified data about all published results that have been confirmed and discredited.

[1] obligatory xkcd: https://www.xkcd.com/882/

[2] for the youngsters: https://en.wikipedia.org/wiki/List_of_minor_The_Hitchhiker%2...

Re: The problem with p-values

#17

Post by vmuhonen » Wed, Oct 12, 2016, 8:07 AM UTC

bayeslives wrote: post #14

jsprogrammer wrote: post #11

>The problem is that the p-value gives the right answer to the wrong question. What we really want to know is not the probability of the observations given a hypothesis about the existence of a real effect, but rather the probability that there is a real effect – that the hypothesis is true – given the observations. And that is a problem of induction. The problem of induction is real and unavoidable in the general ca…

Bayes did not "solve induction". If we define induction as telling which specific model generated this data: that is not possible. Countless models could in theory generate our data. What we need is some restriction. Like a prior. And when was the last time you started a research project without any idea what to expect? And if you did, wouldn't it be wiser to do some literature study, experts interviews etc. before s…

P-value being defined as the probability of observing a result equally or more extreme under a model H0. So if you start with with assumption that H0 is true, there's not much you can say about alternative hypotheses.

The American Statistical Association actually put out a statement this year on the issue of p-value. You can find the whole article here http://dx.doi.org/10.1080/00031305.2016.1154108 but here are the main points:

1) P-values can indicate how incompatible the data are with a specified statistical model.

2) P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

3) Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.

4) Proper inference requires full reporting and transparency.

5) A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

6) By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

As an additional curiosity, the group of writers was not completely unconflicted coming up those definitions and the article contains a number of supplemental articles by the individual authors to clarify/dispute some of the points made.

[edit] fixed formatting

Re: The problem with p-values

#18

Post by taejo » Wed, Oct 12, 2016, 8:10 AM UTC

ikeboy wrote: post #10

ramidarigaz wrote: post #9

Earlier quoted context omitted.

I think there is a good option for choosing your priors in the Earth/Sun problem you mentioned. Just put them at 50/50. That's basically stating that you have no preference for one theory over the other, and it allows for any evidence you gather to be maximally effective in computing the posterior.

That quickly leads to inconsistencies. What's the probability of the earth going around some object bigger than it? What's the probability of the earth going around something bigger than it that isn't the sun? The answer to all 3 cannot all be 50/50.

That's true, but if we assume the probabilities of "Earth orbits Sun" and "Sun orbits Earth" we still get the correct conditional probabilities under the condition "either the Sun orbits the Earth or the Earth orbits the Sun", and the ratio between the two conditional probabilities is the same as the ratio between the absolute probabilities. So we can still decide which hypothesis is better, and by how much.

Re: The problem with p-values

#19

Post by bayeslives » Wed, Oct 12, 2016, 10:39 AM UTC

vmuhonen wrote: post #17

bayeslives wrote: post #14

Earlier quoted context omitted.

Bayes did not "solve induction". If we define induction as telling which specific model generated this data: that is not possible. Countless models could in theory generate our data. What we need is some restriction. Like a prior. And when was the last time you started a research project without any idea what to expect? And if you did, wouldn't it be wiser to do some literature study, experts interviews etc. before s…

P-value being defined as the probability of observing a result equally or more extreme under a model H0. So if you start with with assumption that H0 is true, there's not much you can say about alternative hypotheses. The American Statistical Association actually put out a statement this year on the issue of p-value. You can find the whole article here http://dx.doi.org/10.1080/00031305.2016.1154108 but here are the…

Good points. The NHST thing was invented by Neyman & Pearson as a tool for decision making, not for finding the truth.95% confidence means your intervals will be not to far off, in 95% of all samples.

Perhaps this is nice fur Quality Asurance in factories. Where I do repeated measurements and where I want a simple YES or NO.

But science usually asks: "what can I learn from this specific data? I don't do 100 samples and I'm not interested in being "not to far off moist of the time". I want a best estimation based on this specific sample".

NHST does not give that answer. Bayes does.

Re: The problem with p-values

#20

Post by jsprogrammer » Wed, Oct 12, 2016, 4:07 PM UTC

bayeslives wrote: post #19

vmuhonen wrote: post #17

Earlier quoted context omitted.

P-value being defined as the probability of observing a result equally or more extreme under a model H0. So if you start with with assumption that H0 is true, there's not much you can say about alternative hypotheses. The American Statistical Association actually put out a statement this year on the issue of p-value. You can find the whole article here http://dx.doi.org/10.1080/00031305.2016.1154108 but here are the…

Good points. The NHST thing was invented by Neyman & Pearson as a tool for decision making, not for finding the truth.95% confidence means your intervals will be not to far off, in 95% of all samples. Perhaps this is nice fur Quality Asurance in factories. Where I do repeated measurements and where I want a simple YES or NO. But science usually asks: "what can I learn from this specific data? I don't do 100 samples a…

Science only asks, "Have I observed something contrary to my theories?" For non-deductive theories, the only real approach is to count the number of times you observed something agreeing with your theories vs. the total number of times you observed something (ie. p-value).