Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Just 11% of 53 cancer research papers were reproducible

nature.com

Post reply on HN
Print view View original

161–170 of 184 posts

Re: Just 11% of 53 cancer research papers were reproducible

#161

Post by podperson »

Avshalom wrote: post #125
spikels wrote: post #109

Earlier quoted context omitted.

If its not reproducible it is not science.

Huge swaths of astronomy are functionally unreproducible. We can argue about the math but many phenomena exist as a single example and/or that's basically static on our time scales. The best we can do is see if the math seems to produce similar looking structures when (sparsely) simulated.

I think this is a straw man. Astronomers, astrophysicists, etc. go to enormous lengths to address these issues over time and are well aware of the shortcomings of their work. When black holes were predicted, none had been observed. I'd suggest that astronomy is a terrible place to make claims about irreproducibility.

Cosmology on the other hand...

Re: Just 11% of 53 cancer research papers were reproducible

#163

Post by gwern »

zipdog wrote: post #49

What is more troubling is not that so few of these results results are reproducible, but that it appears almost no-one is trying to reproduce the results of earlier studies. The ability of the scientists who wrote the paper to even get access to the resources necessary to try and reproduce the results is limited. I'm reminded of Feymann's Cargo Cult Science: "I was shocked to hear of an experiment done at the big acc…

Yes, it's a good piece, but actually the part you should've quoted was the part on the mice experiments...

Re: Just 11% of 53 cancer research papers were reproducible

#164

Post by gwern »

kailuowang wrote: post #19
codeulike wrote: post #12

Most published scientific research papers are wrong, according to a new analysis. http://www.newscientist.com/article/dn7915-most-scientific-p... Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 http://dx.doi.org/10.1371/journal.pmed.0020124

Is this analysis a published scientific research?

Depends. By and large, Ioannidis's paper is pointing out the basic consequences of the alpha and beta parameters in Neyman-Pearson hypothesis testing, so there's a very good argument that this paper is merely statistics, and statistics is generally considered to be mathematics, and mathematics not science but its own thing.

Re: Just 11% of 53 cancer research papers were reproducible

#165

Post by gwern »

JDDunn9 wrote: post #149

Perhaps part of this is statistical errors and selection bias. Most studies are done with a 95% confidence interval, but only studies that reject the null hypothesis are published. If we assume only 20% of all experiments reject the null hypothesis, then we should expect ~25% of those to be fasle positives. Journals might also be biased to include more sensationalistic results, which are more likely to be statistical…

> My guess is that if you included all test results, that reject the null hypothesis or not, and are published or not, then the reproducibility rate would be closer to 90%-95%.

No; you need to bring in power, not just assume an alpha of 0.05 and a base-rate of real results of 20%. (Consider the recent neuroscience paper estimating the experiments average a power of like 0.3...)

Re: Just 11% of 53 cancer research papers were reproducible

#166

Post by anologwintermut »

enraged_camel wrote: post #138
jacoblyles wrote: post #104

Earlier quoted context omitted.

I don't understand why science should be a closed guild in the first place.

Because these days it is institutionalized, and in our society that means it is funded by capitalistic entities, i.e. corporations. In other words, the profit motive dictates that making money to enrich a few individuals is more noble than sharing knowledge for the good of society.

Science is funded entirely by corporations? Tell me, where can I buy stock in the NSF, DARPA, NIH? Or where can I find the articles of incorporation for the MacArthur Foundation or any of the other non-profits that also fund research. Or which corporation is paying my PhD stipend --- I really ought to short their stock since they were dumb enough to hire me.

Re: Just 11% of 53 cancer research papers were reproducible

#167

Post by anologwintermut »

zeteo wrote: post #93
jballanc wrote: post #67

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being comp…

>Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev. That's very optimistic, but not necessarily true (even ignoring the question of when ). Priestley succeeded so well because he developed simple, reproducible experiments (expose mercury oxide to sunlight to produce oxygen etc.) that he was happy to share for no financial gain (even traveling to others' labs to help them reproduce his w…

What do paywalls possible have to do with the reproducibility of results? Anyone who has access to a lab capable of even attempting to reproduce such research is either 1) at an institution that has access to those journals or 2) can use google to find the papers anyway. Hopefully both.

The fact that tax-payer funded research is put behind paywalls is a travesty , but it seems wrong to claim it actually hinders researchers from doing anything given.

Re: Just 11% of 53 cancer research papers were reproducible

#168

Post by return0 »

anologwintermut wrote: post #167
zeteo wrote: post #93

Earlier quoted context omitted.

>Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev. That's very optimistic, but not necessarily true (even ignoring the question of when ). Priestley succeeded so well because he developed simple, reproducible experiments (expose mercury oxide to sunlight to produce oxygen etc.) that he was happy to share for no financial gain (even traveling to others' labs to help them reproduce his w…

What do paywalls possible have to do with the reproducibility of results? Anyone who has access to a lab capable of even attempting to reproduce such research is either 1) at an institution that has access to those journals or 2) can use google to find the papers anyway. Hopefully both. The fact that tax-payer funded research is put behind paywalls is a travesty , but it seems wrong to claim it actually hinders resea…

If they were open, you would have an online service that could automatically point contradicting studies, and even make suggestions.

Re: Just 11% of 53 cancer research papers were reproducible

#169

Post by kanzure »

anologwintermut wrote: post #167
zeteo wrote: post #93

Earlier quoted context omitted.

>Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev. That's very optimistic, but not necessarily true (even ignoring the question of when ). Priestley succeeded so well because he developed simple, reproducible experiments (expose mercury oxide to sunlight to produce oxygen etc.) that he was happy to share for no financial gain (even traveling to others' labs to help them reproduce his w…

What do paywalls possible have to do with the reproducibility of results? Anyone who has access to a lab capable of even attempting to reproduce such research is either 1) at an institution that has access to those journals or 2) can use google to find the papers anyway. Hopefully both. The fact that tax-payer funded research is put behind paywalls is a travesty , but it seems wrong to claim it actually hinders resea…

    > What do paywalls possible have to do with the reproducibility
    > of results? Anyone who has access to a lab
Actually, there's no institution subscribed to every journal. There are, in fact, many papers inaccessible to even the most funded labs. Spotty coverage.

Re: Just 11% of 53 cancer research papers were reproducible

#170

Post by yaok »

anologwintermut wrote: post #166
enraged_camel wrote: post #138

Earlier quoted context omitted.

Because these days it is institutionalized, and in our society that means it is funded by capitalistic entities, i.e. corporations. In other words, the profit motive dictates that making money to enrich a few individuals is more noble than sharing knowledge for the good of society.

Science is funded entirely by corporations? Tell me, where can I buy stock in the NSF, DARPA, NIH? Or where can I find the articles of incorporation for the MacArthur Foundation or any of the other non-profits that also fund research. Or which corporation is paying my PhD stipend --- I really ought to short their stock since they were dumb enough to hire me.

Those are government entities, and the government's parent company is the corporatocracy that owns all the politicians and media.

Science is a business in the service of global capitalism. Truth hurts.

I will give you the MacArthur foundation and certain other non-profits. But most non-profits are also operated by the corporatocratic elite, like everything else on this planet.

Post reply on HN