Just 11% of 53 cancer research papers were reproducible • Hacker News Board

Re: Just 11% of 53 cancer research papers were reproducible

#91

Post by pesenti » Thu, May 09, 2013, 3:48 PM UTC

return0 wrote: post #36

pesenti wrote: post #32

The problem is not that research is not reproducible. Researchers are humans and full of biases. The problem is too much emphasis is put on novelty vs. reproducibility. It took a long time for software engineers to recognize the value of testing/validating vs. creating new code. In the same way the research world needs to put new emphasis on reproducing/validating results.

The problem is that too much emphasis is put on prestige and reputation than on actual impact of the work. Coming from a physics background I always felt that life scientists tend to be overly audacious in their claims, am I alone in this?

Is it only in life sciences? Remember the cold fusion and the faster than light particle fiascos?

Re: Just 11% of 53 cancer research papers were reproducible

#92

Post by micro_cam » Thu, May 09, 2013, 3:52 PM UTC

RyanMcGreal wrote: post #2

The analysis in question: http://www.nature.com/nature/journal/v483/n7391/full/483531a...

This is a really insightful article that contains a number of points that qualify the result and present lessons for any one interested in data science.

A few points that stand out to me:

!) The 11% number comes not from a randomized sample of cancer papers but from attempts made by AmGen to reproduce results that might be useful in drug development. This means that they made good effort to reproduce these results but there is also a strong selection bias that needs to be acknowledged.

2) They point out that using survival time as the measurement complicates things. I've done a lot of machine learning and statistics on cancer and medical data and, in my experience, this can not be overemphasized. and There are loads of confounding factors that contribute to survival time. I expect that big break throughs will come as we develop rigorous ways for measuring the behavior of a tumor (does it metastasize? how does it feed itself?) and use those as the targets of our regression. (Currently these thins are measured by human visual inspection if at all.)

3) They point out that the studies that were repeatable were the ones that were careful about using blinded controls, and eliminating investigator bias. This is basic stuff but easy to overlook. In terms applicable to a startup, a data scientist needs to me motivated to vet existing and proposed practices and identify flawed ones as much or more then they are motivated to maximize gain.

Re: Just 11% of 53 cancer research papers were reproducible

#93

Post by zeteo » Thu, May 09, 2013, 4:01 PM UTC

jballanc wrote: post #67

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being comp…

>Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev.

That's very optimistic, but not necessarily true (even ignoring the question of when). Priestley succeeded so well because he developed simple, reproducible experiments (expose mercury oxide to sunlight to produce oxygen etc.) that he was happy to share for no financial gain (even traveling to others' labs to help them reproduce his work [1]). The current environment of paywalls and competitive grants seems hardly conducive to the rise of similar figures.

[1] http://en.wikipedia.org/wiki/Experiments_and_Observations_on...

Re: Just 11% of 53 cancer research papers were reproducible

#94

Post by Florin_Andrei » Thu, May 09, 2013, 4:02 PM UTC

jballanc wrote: post #67

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being comp…

So, basically, it's a combination between: A) A really hard problem to solve; and B) A heck of a lot of pressure to get it solved.

Re: Just 11% of 53 cancer research papers were reproducible

#95

Post by kailuowang » Thu, May 09, 2013, 4:07 PM UTC

vixen99 wrote: post #82

kailuowang wrote: post #18

Earlier quoted context omitted.

The possibility of CO2 causing global warming isn't 100% but it isn't very low either. It is the risk we are talking about, the risk of doing nothing and let the man made green house (if there is one) turn earth into an irreversible disastrous environment. Most CO2 emitting energy sources are not sustainable anyway and many of them emit other proved pollution as well. There is really nothing lost to going green energ…

Without qualification, I'm afraid what you're saying is parlously close to mere verbiage. Even the most ardent climate sceptic does not object to solar, wind or water energy under certain conditions. What is 'Going Green' then? In the UK it currently means, for instance, paying wind farms a million pounds sterling a day to produce no energy at all. Meanwhile poor people die because they can't afford to keep themselve…

"Going Green" does not equal to switch to renewable energy at the cost of poor people's lives for gods sake. It simply means that recognizing the global warming and have a plan to switch to renewable energy, for example, take some of the hundreds of billions of dollars of the oil companies' profits to invest on renewable energy.

Re: Just 11% of 53 cancer research papers were reproducible

#96

Post by Someone » Thu, May 09, 2013, 4:08 PM UTC

epistasis wrote: post #45

campnic wrote: post #27

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch t…

This is an entirely different issue than code; code mostly does the same thing when you run it twice. There's no such guarantee in biology. A cancer cell line growing in one lab may behave differently than descendants of those cells in a different lab. This may be due to slight differences in the timings between feeding the cells and the experiments, stochastic responses built into the biology, slight variations betw…

Also, sometimes, doing the experiment is extremely hard. I know a guy who only slightly jokingly claims he got his Ph.D. on one brain cell. He spent a couple of years building a setup to measure electrical activity of neurons, and 'had' one cell for half an hour or so (you stick an electrode in a cell, hope it doesn't die in the process, and then hope your subject animal remains perfectly subdued, and that there will not be external vibrations that make your electrode move, thus losing contact with the cell or killing it)

Reproducible? Many people could do it, if they made the effort, but how long it would take is anybody's guess.

Experiments like that require a lot of fingerspitzengefühl from those performing them. Worse, that doesn't readily translate between labs. For example, an experimental setup in a small lab might force an experimenter in a body posture that makes his hand vibrate less when doing the experiment. If he isn't aware of that advantage, he will not be able to repeat his experiment in a better lab (I also know guys who jokingly stated they got best results with a slight hangover; there might have been some truth in that)

Re: Just 11% of 53 cancer research papers were reproducible

#97

Post by teeja » Thu, May 09, 2013, 4:33 PM UTC

The tradition in science publishing has been to leave out the nitty-gritty details. Most specialists will know (or quickly deduce) most available methods (unless the results are a radical improvement). As for the details, that's what conferences and telephones and joining the department and gossip are for.

Re: Just 11% of 53 cancer research papers were reproducible

#98

Post by [deleted] » Thu, May 09, 2013, 4:42 PM UTC

campnic wrote: post #83

epistasis wrote: post #45

Earlier quoted context omitted.

This is an entirely different issue than code; code mostly does the same thing when you run it twice. There's no such guarantee in biology. A cancer cell line growing in one lab may behave differently than descendants of those cells in a different lab. This may be due to slight differences in the timings between feeding the cells and the experiments, stochastic responses built into the biology, slight variations betw…

Oh, I agree. Biological experiment reproducibility is an incredibly hard problem. You are probably right that it is 'trivial' by comparison in the same way that landing on mars is trivial to landing on Alpha Centauri.

[deleted]

Re: Just 11% of 53 cancer research papers were reproducible

#99

Post by [deleted] » Thu, May 09, 2013, 4:43 PM UTC

codeulike wrote: post #12

Most published scientific research papers are wrong, according to a new analysis. http://www.newscientist.com/article/dn7915-most-scientific-p... Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124 http://dx.doi.org/10.1371/journal.pmed.0020124

[deleted]

Re: Just 11% of 53 cancer research papers were reproducible

#100

Post by ars » Thu, May 09, 2013, 4:49 PM UTC

return0 wrote: post #30

Ensorceled wrote: post #25

Earlier quoted context omitted.

Nature wrote an editorial justifying the situation, you can read it here http://www.nature.com/nature/journal/v485/n7396/full/485041e...

> "those authors required them to sign an agreement that they would not disclose their findings about specific papers" I don't know why this doesn't cause a huge stir in the scientific community. Seems like everyone is fine with people sweeping their negative data under the rug. Shameful is a very mild word for that.

How were those authors able to enforce this request? To be a proper paper it should not be necessary to contact the author in order to reproduce it.

If you had to ask the author for more details then their paper was incomplete - perhaps that's why it could not be reproduced.