Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Just 11% of 53 cancer research papers were reproducible

nature.com

Post reply on HN
Print view View original

151–160 of 184 posts

Re: Just 11% of 53 cancer research papers were reproducible

#151

Post by ttrreeww »

pessimizer wrote: post #74

There's an awful lot of money in cancer (research funding/patents), an awful lot of diet/lifestyle handwaving in cancer research conclusions (that aren't about treatments), and very little progress in making cancer more survivable. Just those facts would lead me to assume that most of the published research is bad. The great part is that 11% of the papers are reproducible, so the science isn't at a standstill.

Cancer is one of those sickness where you either cure it, or you die.

Problem is, for profit companies seeks "treatment" to maximize profit, they don't seek the cure. Hence no progress on cancer.

Re: Just 11% of 53 cancer research papers were reproducible

#152

Post by ttrreeww »

mchusma wrote: post #3

This is a great issue to highlight. However, I'm unconvinced we need to optimize around creating research that can be reproducible, rather that the process of science generates the best possible results. Both the scientific and lay public should read this (appropriately) as: one study does not prove a result, there is an 89% chance that a single research paper is simply wrong. This mentality seems to be the bigger ga…

Cold fusion was reproducible. What are you suggesting we do here?

Re: Just 11% of 53 cancer research papers were reproducible

#153

Post by Mz »

pessimizer wrote: post #74

There's an awful lot of money in cancer (research funding/patents), an awful lot of diet/lifestyle handwaving in cancer research conclusions (that aren't about treatments), and very little progress in making cancer more survivable. Just those facts would lead me to assume that most of the published research is bad. The great part is that 11% of the papers are reproducible, so the science isn't at a standstill.

That isn't limited to cancer. Western medicine is driven by a lot of bad mental and social models. Doctors only make money if you are sick. That strikes me as a seriously flawed model right there.

Re: Just 11% of 53 cancer research papers were reproducible

#154

Post by codex »

This can be explained without bringing in accusations of fraud or incompetence [1].

Basically, if you have a bazillion experiments running and only publish the results which are significant, you still have a huge predictability problem, because most of those statistically significant results will be due to chance alone.

For example, assume that only 1% of cancer experiments produce something valid and interesting, and that standard for statistical significance is the industry standard 95%, or 1 in 20. If 100,000 hypothesis are tested, you'll get 100,000 * 1% = 1000 valid results and 100,000 * 5% = 5000 results from chance alone. In this example, only 16% of the published results will be reproducible, because only 1000 / 6000 are valid.

[1] http://sciencehouse.wordpress.com/2009/05/08/why-most-publis...

Re: Just 11% of 53 cancer research papers were reproducible

#155

Post by momerath »

cowsandmilk wrote: post #90
Thrymr wrote: post #72

Earlier quoted context omitted.

Recent article on git and reproducability in science: http://www.scfbm.org/content/8/1/7 It is badly needed.

That article says "Data are ideal for managing with Git." I one time tried using git to manage my data. The problem is, I frequently have thousands of files and gigabytes of data. And git just does not handle that well.[1] One time, I even tried building a git repo that just had the history of pdb snapshots. The PDB frequently has updates, and I have run into many cases where an analysis of a structure was done in a…

You might find git-annex useful:

http://git-annex.branchable.com/

Re: Just 11% of 53 cancer research papers were reproducible

#157

Post by specialist »

jerf wrote: post #66
zipdog wrote: post #49

What is more troubling is not that so few of these results results are reproducible, but that it appears almost no-one is trying to reproduce the results of earlier studies. The ability of the scientists who wrote the paper to even get access to the resources necessary to try and reproduce the results is limited. I'm reminded of Feymann's Cargo Cult Science: "I was shocked to hear of an experiment done at the big acc…

If I had to sum up science in one phrase, I wouldn't say anything about the "scientific process" or anything like that. I would say: "Look for reasons why you are wrong , not reasons why you are right." You can always find reasons why you're right. You can always do an astrology forecast and find someone for whom it was dead-on accurate. That's not the problem with astrology, the problem lies in how often it is wrong…

"Look for reasons why you are wrong, not reasons why you are right."

I share your worldview. Makes intuitive sense to me. It's intellectually honest. And if I'm wrong about something and no one corrects me, I get kinda grumpy.

Maybe everyone else already knows, but I learned relatively late that it's called Popperian.

http://en.wikipedia.org/wiki/Karl_Popper#Philosophy_of_scien... http://en.wikipedia.org/wiki/Falsifiability

Re: Just 11% of 53 cancer research papers were reproducible

#158

Post by zmmmmm »

Cancer is such a weird beast that I'm actually not as alarmed by this as it might seem reasonable to be.

Only in the last year or so people have realized that how heterogeneous cancer is. A patient biopsied 10 times from different parts of the cancer turns out to have 10 different (but related) cancers. While such major findings are still occurring about the underlying nature of cancer, it is not surprising that studies are hard to produce; they don't even know which parameters need to be controlled yet to make them reproducible.

Re: Just 11% of 53 cancer research papers were reproducible

#159

Post by jerf »

specialist wrote: post #157
jerf wrote: post #66

Earlier quoted context omitted.

If I had to sum up science in one phrase, I wouldn't say anything about the "scientific process" or anything like that. I would say: "Look for reasons why you are wrong , not reasons why you are right." You can always find reasons why you're right. You can always do an astrology forecast and find someone for whom it was dead-on accurate. That's not the problem with astrology, the problem lies in how often it is wrong…

"Look for reasons why you are wrong, not reasons why you are right." I share your worldview. Makes intuitive sense to me. It's intellectually honest. And if I'm wrong about something and no one corrects me, I get kinda grumpy. Maybe everyone else already knows, but I learned relatively late that it's called Popperian. http://en.wikipedia.org/wiki/Karl_Popper#Philosophy_of_scien... http://en.wikipedia.org/wiki/Falsifi…

Falsifiability is one natural landing point, but it is also somewhat controversial. What I'm advocating isn't so much a philosophy as a state of mind, one hopefully less controversial than trying to declare a "definition of science". I think of it more like a mind hack you can perform on yourself. (So much discipline boils down to figuring out how your conscious brain can fool your subconscious brain.)
Post reply on HN