Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

blog.scienceexchange.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#31

Post by cowsandmilk »

Alex3917 wrote: post #27
tokenadult wrote: post #22

Earlier quoted context omitted.

You haven't validated any of those assumptions. Moreover a "common scientific mistake," while regrettable and something to reduce, may not have a lot of impact on actual clinical practice based on what physicians read in published medical journal articles. You'll have to connect a lot more dots to show that there is a high-impact problem here. Meanwhile, life expectancy at age 40, at age 60, and at even higher ages i…

> You haven't validated any of those assumptions. There's an entire field of researchers researching researchers, so it's trivially easy to put together a list of common research errors and their prevalence. The only difficult thing is calculating the degree of independence. But there isn't any evidence, so far as I know, that these errors are highly correlated. Nor is there much reason to beleive that they are, sinc…

> But as I've posted previously, and as you can find trivially easily via Google, of the 30 year increase in life expectancy in the US, a minimum of 25 of those years have come from public health improvements rather than from modern medicine.

Assuming you are getting the 25 years from the CDC[1], you are being obtuse. Immunizations is the first thing listed. Vaccines are a component of modern medicine. Control of Infectious diseases includes antibiotics, which are medicine. Clicking on "Declines in Deaths from Heart Disease and Stroke" shows that it includes "an increase in the percentage of persons with hypertension who have the condition treated and controlled" using pharmaceuticals, definitively modern medicine.

In summary, these 25 years aren't coming from just getting people to quit smoking. There's a large component that is modern medicine.

[1] http://www.cdc.gov/about/history/tengpha.htm

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#32

Post by bmahmood »

Fomite wrote: post #24
mathattack wrote: post #9

Earlier quoted context omitted.

If $1.3M can cover 50, they should fund reproducing 500, or even 5000. I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.

Beyond it being expensive to replicate once, for observational studies and clinical trials, you need many repetitions because of statistical variability. That gets expensive in a hurry.

You are right that clinical trial studies would be much more time intensive and costly. The studies we will be replicating through the Reproducibility Initiative are more preclinical cancer studies, rather than later-stage clinical trials.

We felt the preclinical stage was more important to validate, as much of the research is based on academic studies with over 60% failure rate for replication (see http://www.nature.com/nature/journal/v483/n7391/full/483531a...)

By clinical trials, a lot of the pharma companies have already done initial validation, so extra validation is redundant. We hope a reproducibility system to validate preclinical research can help patient groups, foundations, and industry better identify reproducible oncology targets, and hence the focus on preclinical studies (which in turn are less expensive).

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#33

Post by Alex3917 »

cowsandmilk wrote: post #31
Alex3917 wrote: post #27

Earlier quoted context omitted.

> You haven't validated any of those assumptions. There's an entire field of researchers researching researchers, so it's trivially easy to put together a list of common research errors and their prevalence. The only difficult thing is calculating the degree of independence. But there isn't any evidence, so far as I know, that these errors are highly correlated. Nor is there much reason to beleive that they are, sinc…

> But as I've posted previously, and as you can find trivially easily via Google, of the 30 year increase in life expectancy in the US, a minimum of 25 of those years have come from public health improvements rather than from modern medicine. Assuming you are getting the 25 years from the CDC[1], you are being obtuse. Immunizations is the first thing listed. Vaccines are a component of modern medicine. Control of Inf…

"Vaccines are a component of modern medicine."

Given that the argument was about the reliability of modern medical studies, it would probably be only reasonable to include studies that came onto the market after the era of plabeco controlled RCTs. But many if not most of the most impactful vaccines came out before then. Of the remaining important vaccines, they account for only a tiny fraction of the literature.

But more importantly, it's not even a valid argument to begin with since the increase in life expectancy has nothing to do with the percentage of medical papers that are flawed.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#34

Post by Fomite »

bmahmood wrote: post #32
Fomite wrote: post #24

Earlier quoted context omitted.

Beyond it being expensive to replicate once, for observational studies and clinical trials, you need many repetitions because of statistical variability. That gets expensive in a hurry.

You are right that clinical trial studies would be much more time intensive and costly. The studies we will be replicating through the Reproducibility Initiative are more preclinical cancer studies, rather than later-stage clinical trials. We felt the preclinical stage was more important to validate, as much of the research is based on academic studies with over 60% failure rate for replication (see http://www.nature…

Agreed - and this is a hugely important effort. This is more me musing about replication vs. reproducibility (on the computational/statistical end) vs. repeatability.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#35

Post by Alex3917 »

carbocation wrote: post #30
Alex3917 wrote: post #27

Earlier quoted context omitted.

> You haven't validated any of those assumptions. There's an entire field of researchers researching researchers, so it's trivially easy to put together a list of common research errors and their prevalence. The only difficult thing is calculating the degree of independence. But there isn't any evidence, so far as I know, that these errors are highly correlated. Nor is there much reason to beleive that they are, sinc…

> But as I've posted previously, and as you can find trivially easily via Google, of the 30 year increase in life expectancy in the US, a minimum of 25 of those years have come from public health improvements rather than from modern medicine. Do you categorize vaccination as modern medicine, or as public health? Is there no overlap, some overlap, or much overlap in the concepts of "modern medicine" and "public health…

> Do you categorize vaccination as modern medicine, or as public health?

It depends on what point you're trying to make. E.g. if you're talking about contribution of pharmaceutical drugs to longevity, then it doesn't make sense to include vaccines since they're not drugs. If you're talking about the contributions of modern medicine as a whole, then it make sense to include all the vaccines developed roughly since Dr. Gold popularized the gold standard clinical trials, so sometime in the mid to late 1950s. Or if you're talking about the contributions of the western medical system as a whole since its inception, then it probably makes sense to include all vaccines, etc.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#36

Post by doctoboggan »

This is awesome. I met some of the people behind The Center For Open Science at SciPy this year. They seemed very passionate. I hope the idea of reproducing experiments as a matter of course becomes more common. Maybe in the future to be a reputable scientist you will have had to reproduce many of the current experiments of the time.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#37

Post by dwaltrip »

Alex3917 wrote: post #20
napoleoncomplex wrote: post #16

Reproducibility in science is something that badly needs this push. It's an incredibly difficult "sell" to anyone with funds for research, and I'm extremely happy that they've found capital for it. The foundations of our scientific knowledge need to be solidified, and from all the science news and developments, this one is the one that makes me by far the most excited for the future of science. Next on the list, open…

Assuming there are about 3 dozen common scientific mistakes in medical journal articles, each appearing in on average 10-20% of papers with a relatively high degree of independence, there isn't really much foundation for medical science to begin with. (And I think these assumptions are probably fairly accurate.)

The point of your comment isn't clear. Should everyone go home and give up then? This initiative surely alleviates some of your concerns, yes? And while I don't have too much specific knowledge, my general sense is that our medical knowledge has increased enormously in the past 100 years, and that this trend seems primed to continue. But I'm generally pretty optimistic.

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#38

Post by Create »

irollboozers wrote: post #5

Science Exchange is leading and pushing ahead with this very important work. They are addressing what the public funders and private industry can't and won't do, but at scale this becomes really powerful. Great stuff.

They did it publicly, ie

Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124

and privately: the matter of fact is, pharma companies are unhappy, because according to them, over 70% is irreproducible (euphemism for BS).

This is why ie. BMJ will only accept papers from January on that publish evidence ie. data also. One wonders what happened before...

BMJ 2013; 347 doi: http://dx.doi.org/10.1136/bmj.f5975

Re: Reproducibility Initiative gets $1.3M grant to validate 50 cancer studies

#39

Post by [deleted] »

Osmium wrote: post #19
mathattack wrote: post #9

Earlier quoted context omitted.

If $1.3M can cover 50, they should fund reproducing 500, or even 5000. I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. If you're getting into medicine, it's more expensive because you may need to reproduce trials. Psychology even more so because the tests are so soft. But of course psych and medical studies are the ones most in need of replication.

> I suspect that many pure science studies are relatively easier to reproduce since it's more about basic properties. Not necessarily true at all actually. To use an example, the field I work in requires that we grow our own semiconductor crystals. Different growth reactors in different universities producing nominally the exact same substance will result in sometimes vastly different properties of the end result whe…

[deleted]
Post reply on HN