Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies • Hacker News Board

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#21

Post by yummyfajitas » Wed, Oct 16, 2013, 10:05 PM UTC

mjn wrote: post #8

gabemart wrote: post #6

I don't know very much about how reproducibility validation works. Is it the case that, if we assume p= ~0.05 and all 50 original studies are perfect, we would expect the first iteration of reproducibility validation to fail for ~2 of the 50 studies?

Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really just by chance. Therefore, if we were now given the real ground truth, it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. But you seem to be positing the reverse: a hypoth…

it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all.

Not quite. Assume P of the tests are truly positive, and N are truly negative.

We expect to see approximately 0.05 x N + A x P positives in the entire sample, where A is the probability of a false negative. So the fraction of true positives not replicated is likely to be 0.05 x N / (0.05 x N + A x P).

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#22

Post by tokenadult » Wed, Oct 16, 2013, 10:18 PM UTC

Alex3917 wrote: post #20

napoleoncomplex wrote: post #16

Reproducibility in science is something that badly needs this push. It's an incredibly difficult "sell" to anyone with funds for research, and I'm extremely happy that they've found capital for it. The foundations of our scientific knowledge need to be solidified, and from all the science news and developments, this one is the one that makes me by far the most excited for the future of science. Next on the list, open…

Assuming there are about 3 dozen common scientific mistakes in medical journal articles, each appearing in on average 10-20% of papers with a relatively high degree of independence, there isn't really much foundation for medical science to begin with. (And I think these assumptions are probably fairly accurate.)

You haven't validated any of those assumptions. Moreover a "common scientific mistake," while regrettable and something to reduce, may not have a lot of impact on actual clinical practice based on what physicians read in published medical journal articles. You'll have to connect a lot more dots to show that there is a high-impact problem here.

Meanwhile, life expectancy at age 40, at age 60, and at even higher ages is still rising throughout the developed countries of the world,[1] so at least the trend lines look good for continued improvements in health as medical research is incrementally improved. Girls born since 2000 in the developed world are more likely than not to reach the age of 100, with boys likely to enjoy lifespans almost as long. The article "The Biodemography of Human Ageing"[2] by James Vaupel, originally published in the journal Nature in 2010, is a good current reference on the subject. Vaupel is one of the leading scholars on the demography of aging and how to adjust for time trends in life expectancy. His striking finding is "Humans are living longer than ever before. In fact, newborn children in high-income countries can expect to live to more than 100 years. Starting in the mid-1800s, human longevity has increased dramatically and life expectancy is increasing by an average of six hours a day."[3] An article in a series on Slate, "Why Are You Not Dead Yet? Life expectancy doubled in past 150 years. Here’s why"[4] makes clear that it's progress here, and progress there, and a lot of little things adding up that have resulted in the dramatic increases in healthy lifespan over the last few generations.

[1] http://www.scientificamerican.com/article.cfm?id=longevity-w...

[2] http://www.demographic-challenge.com/files/downloads/2eb51e2...

[3] http://www.prb.org/Journalists/Webcasts/2010/humanlongevity....

[4] http://www.slate.com/articles/health_and_science/science_of_...

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#23

Post by [deleted] » Wed, Oct 16, 2013, 10:24 PM UTC

yummyfajitas wrote: post #21

mjn wrote: post #8

Earlier quoted context omitted.

Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really just by chance. Therefore, if we were now given the real ground truth, it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. But you seem to be positing the reverse: a hypoth…

it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. Not quite. Assume P of the tests are truly positive, and N are truly negative. We expect to see approximately 0.05 x N + A x P positives in the entire sample, where A is the probability of a false negative. So the fraction of true positives not replicated is likely to be 0.05…

[deleted]

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#24

Post by Fomite » Wed, Oct 16, 2013, 10:24 PM UTC

mathattack wrote: post #9

Osmium wrote: post #7

Honestly, I'm impressed $1.3M is enough for 50 studies! Though, of course, verification should be cheaper than the original research since you know exactly what to look for and how to find it.

If $1.3M can cover 50, they should fund reproducing 500, or even 5000. I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.

Beyond it being expensive to replicate once, for observational studies and clinical trials, you need many repetitions because of statistical variability. That gets expensive in a hurry.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#25

Post by gwern » Wed, Oct 16, 2013, 10:25 PM UTC

mjn wrote: post #8

gabemart wrote: post #6

I don't know very much about how reproducibility validation works. Is it the case that, if we assume p= ~0.05 and all 50 original studies are perfect, we would expect the first iteration of reproducibility validation to fail for ~2 of the 50 studies?

Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really just by chance. Therefore, if we were now given the real ground truth, it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. But you seem to be positing the reverse: a hypoth…

> That depends on several things about the replication attempt, such as sample size and methodology, summarized into the concept of statistical power: http://en.wikipedia.org/wiki/Statistical_power

It's not just power, you also need prior odds if you want to make an unconditional estimate. This is pretty much the core of Ioannides's famous paper http://www.plosmedicine.org/article/info:doi/10.1371/journal... - even after taking power into account, the replication rate could be anywhere from 0 to say 80% depending on the prior odds.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#26

Post by gwern » Wed, Oct 16, 2013, 10:27 PM UTC

gabemart wrote: post #11

mjn wrote: post #8

Earlier quoted context omitted.

Not necessarily 5%, no. The p-value is controlling the false-positive rate, i.e. it constrains the likelihood that a seemingly-significant result was really just by chance. Therefore, if we were now given the real ground truth, it's likely that ~5% of the original positive results (if the studies were properly conducted) would turn out to be unsubstantiated after all. But you seem to be positing the reverse: a hypoth…

Thanks, very interesting. It was silly of me to assume that the p-value could be used in that way when, as you explain, it's not measuring or indicating that at all.

Don't feel too bad. It's a very common mistake; see some of the citations in https://en.wikipedia.org/wiki/P-value#Misunderstandings and http://lesswrong.com/lw/g13/against_nhst/

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#27

Post by Alex3917 » Wed, Oct 16, 2013, 10:35 PM UTC

tokenadult wrote: post #22

Alex3917 wrote: post #20

Earlier quoted context omitted.

Assuming there are about 3 dozen common scientific mistakes in medical journal articles, each appearing in on average 10-20% of papers with a relatively high degree of independence, there isn't really much foundation for medical science to begin with. (And I think these assumptions are probably fairly accurate.)

You haven't validated any of those assumptions. Moreover a "common scientific mistake," while regrettable and something to reduce, may not have a lot of impact on actual clinical practice based on what physicians read in published medical journal articles. You'll have to connect a lot more dots to show that there is a high-impact problem here. Meanwhile, life expectancy at age 40, at age 60, and at even higher ages i…

> You haven't validated any of those assumptions.

There's an entire field of researchers researching researchers, so it's trivially easy to put together a list of common research errors and their prevalence. The only difficult thing is calculating the degree of independence. But there isn't any evidence, so far as I know, that these errors are highly correlated. Nor is there much reason to beleive that they are, since presumably the more errors a study has, the less likely it is to be published. Also, we already know from Ioannnidis et al. that the percentage of medical studies that can't be replicated is (iirc) in the 80-90% range. And just because a study can be replicated doesn't mean it's accurate, so 10-20% is really an upper bound on accuracy.

> Meanwhile, life expectancy at age 40, at age 60, and at even higher ages is still rising throughout the developed countries of the world

But as I've posted previously, and as you can find trivially easily via Google, of the 30 year increase in life expectancy in the US, a minimum of 25 of those years have come from public health improvements rather than from modern medicine.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#28

Post by mathattack » Wed, Oct 16, 2013, 10:35 PM UTC

Osmium wrote: post #19

mathattack wrote: post #9

Earlier quoted context omitted.

If $1.3M can cover 50, they should fund reproducing 500, or even 5000. I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.

> I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. Not necessarily true at all actually. To use an example, the field I work in requires that we grow our own semiconductor crystals. Different growth reactors in different universities producing nominally the exact same substance will result in sometimes vastly different properties of the end result whe…

Indeed - thank you for clarifying. This does sound expensive, though all the more necessary.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#29

Post by [deleted] » Wed, Oct 16, 2013, 10:52 PM UTC

tokenadult wrote: post #22

Alex3917 wrote: post #20

Earlier quoted context omitted.

Assuming there are about 3 dozen common scientific mistakes in medical journal articles, each appearing in on average 10-20% of papers with a relatively high degree of independence, there isn't really much foundation for medical science to begin with. (And I think these assumptions are probably fairly accurate.)

You haven't validated any of those assumptions. Moreover a "common scientific mistake," while regrettable and something to reduce, may not have a lot of impact on actual clinical practice based on what physicians read in published medical journal articles. You'll have to connect a lot more dots to show that there is a high-impact problem here. Meanwhile, life expectancy at age 40, at age 60, and at even higher ages i…

[deleted]

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#30

Post by carbocation » Wed, Oct 16, 2013, 10:54 PM UTC

Alex3917 wrote: post #27

tokenadult wrote: post #22

Earlier quoted context omitted.

You haven't validated any of those assumptions. Moreover a "common scientific mistake," while regrettable and something to reduce, may not have a lot of impact on actual clinical practice based on what physicians read in published medical journal articles. You'll have to connect a lot more dots to show that there is a high-impact problem here. Meanwhile, life expectancy at age 40, at age 60, and at even higher ages i…

> You haven't validated any of those assumptions. There's an entire field of researchers researching researchers, so it's trivially easy to put together a list of common research errors and their prevalence. The only difficult thing is calculating the degree of independence. But there isn't any evidence, so far as I know, that these errors are highly correlated. Nor is there much reason to beleive that they are, sinc…

> But as I've posted previously, and as you can find trivially easily via Google, of the 30 year increase in life expectancy in the US, a minimum of 25 of those years have come from public health improvements rather than from modern medicine.

Do you categorize vaccination as modern medicine, or as public health? Is there no overlap, some overlap, or much overlap in the concepts of "modern medicine" and "public health"?