Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
blog.scienceexchange.com
Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
1–10 of 39 posts
Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
#2this is really phenomenal. Congratulations, SE.
Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
#3awesome congrats guys!
Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
#4Congratulations SciEx !
Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
#5Science Exchange is leading and pushing ahead with this very important work. They are addressing what the public funders and private industry can't and won't do, but at scale this becomes really powerful. Great stuff.
Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
#6I don't know very much about how reproducibility validation works. Is it the case that, if we assume p= ~0.05 and all 50 original studies are perfect, we would expect the first iteration of reproducibility validation to fail for ~2 of the 50 studies?
Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
#7Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course, verification should be cheaper than the original research since you know exactly what to look for and how to find it.
Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
#8I don't know very much about how reproducibility validation works. Is it the case that, if we assume p= ~0.05 and all 50 original studies are perfect, we would expect the first iteration of reproducibility validation to fail for ~2 of the 50 studies?
Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really just by chance. Therefore, if we were now given the real ground truth, it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all.
But you seem to be positing the reverse: a hypothetical case where we assume that the results of the original studies were in fact correct (i.e. ground truth), and we want a sort of false-negative rate, the chance that a fact would not be confirmed by a replication study, despite actually being true. That depends on several things about the replication attempt, such as sample size and methodology, summarized into the concept of statistical power: http://en.wikipedia.org/wiki/Statistical_power
The overall failure-to-replicate rate would involve both aspects, with four possible outcomes at varying likelihood: 1) a correct result that was confirmed; 2) a correct result that failed to be confirmed; 3) an incorrect result that was confirmed anyway; and 4) an incorrect result that failed to be confirmed. Obviously it would be ideal if #1 and #4 were much higher than #2 and #3.
Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
#9Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course, verification should be cheaper than the original research since you know exactly what to look for and how to find it.
If $1.3M can cover 50, they should fund reproducing 500, or even 5000.
I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.
Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies
#10I don't know very much about how reproducibility validation works. Is it the case that, if we assume p= ~0.05 and all 50 original studies are perfect, we would expect the first iteration of reproducibility validation to fail for ~2 of the 50 studies?
This is Bilal from Science Exchange, and I help with the Reproducibility Intiative as well.
I think that in general you're right, with the assumption that p=0.05. However, each of the studies being replicated is composed of many intricate experiments. We will be reproducing most of those experiments, so while the odds of a single validation experiment failing are certainly not zero, I think the chances of an entire paper's worth of reproduction experiments being false negatives is much less likely. All of the protocols and reagents used, and the data from the replication studies, will be publicly available when they are complete, so that everyone can see what we did and how we did it.