Just 11% of 53 cancer research papers were reproducible • Hacker News Board

Re: Just 11% of 53 cancer research papers were reproducible

#71

Post by pcrh » Thu, May 09, 2013, 2:00 PM UTC

rdfi wrote: post #63

pcrh wrote: post #59

I work in biomedical research, and this finding has been discussed quite broadly. Most researchers don't believe it. Exactly reproducing novel findings usually requires a significant investment into the underlying procedures, which most places will not undertake. Reproduction instead usually occurs as part of an extension of the initial findings. The complexity of biology means that the first findings are frequently…

Really? Meaning that either there's another unaccounted variable(s) that controls the effect of A in B and C, meaning that we cannot conclude anything from the experiment. Yes and no is not acceptable...

It is exactly as you suggest, additional variables. Bear in mind that a single human cell has ~25,000 genes, and therefore at least that many potential degrees of freedom, even before you include external variables. It is very difficult to control for all of them.

Re: Just 11% of 53 cancer research papers were reproducible

#72

Post by Thrymr » Thu, May 09, 2013, 2:01 PM UTC

campnic wrote: post #27

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch t…

Recent article on git and reproducability in science: http://www.scfbm.org/content/8/1/7

It is badly needed.

Re: Just 11% of 53 cancer research papers were reproducible

#73

Post by epistasis » Thu, May 09, 2013, 2:08 PM UTC

jpdoctor wrote: post #69

jballanc wrote: post #67

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being comp…

> It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.

You're assuming that only "reproducible" results are valid, which is incorrect. Results that are reproducible within a lab, but not when attempted by other people in different settings indicate that there's an unaccounted for variable, and that falls into the 89%. There's still something valuable there, it's just that we don't yet know all the variables.

We could just give up, or we could try to continue study of something incredibly complex. Given that we are gaining some ground, it's clearly not a useless endeavor, it's just an extremely difficult one.

Re: Just 11% of 53 cancer research papers were reproducible

#74

Post by pessimizer » Thu, May 09, 2013, 2:13 PM UTC

There's an awful lot of money in cancer (research funding/patents), an awful lot of diet/lifestyle handwaving in cancer research conclusions (that aren't about treatments), and very little progress in making cancer more survivable.

Just those facts would lead me to assume that most of the published research is bad. The great part is that 11% of the papers are reproducible, so the science isn't at a standstill.

Re: Just 11% of 53 cancer research papers were reproducible

#75

Post by shabble » Thu, May 09, 2013, 2:16 PM UTC

epistasis wrote: post #45

campnic wrote: post #27

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch t…

This is an entirely different issue than code; code mostly does the same thing when you run it twice. There's no such guarantee in biology. A cancer cell line growing in one lab may behave differently than descendants of those cells in a different lab. This may be due to slight differences in the timings between feeding the cells and the experiments, stochastic responses built into the biology, slight variations betw…

Not to mention the ever-present zombie threat[1][2]

[1] https://www.ncbi.nlm.nih.gov/pubmed/10516762?dopt=Citation

[2] https://en.wikipedia.org/wiki/List_of_contaminated_cell_line...

Re: Just 11% of 53 cancer research papers were reproducible

#76

Post by nitid_name » Thu, May 09, 2013, 2:17 PM UTC

campnic wrote: post #27

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch t…

As someone who, fresh out of high school, coded for a quite published astrophysicist at a major government research institution, I can confirm that I had no idea what I was doing.

Re: Just 11% of 53 cancer research papers were reproducible

#77

Post by Retric » Thu, May 09, 2013, 2:19 PM UTC

jpdoctor wrote: post #69

jballanc wrote: post #67

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being comp…

> It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.

If 11% of the papers are reproducible and people are trying to reproduce the results then your still making progress. Just the slow and expencive kind. Considering how complex the subject matter is I don's think it's reasonable to expect anything else.

The problem is people are not trying to reproduce results which harms the field and slows everything down.

Re: Just 11% of 53 cancer research papers were reproducible

#78

Post by kirk21 » Thu, May 09, 2013, 2:27 PM UTC

tokenadult wrote: post #57

This editorial commentary and the article on which it is based are part of an ongoing effort to improve the quality of scientific publication in a number of disciplines. The Retraction Watch group blog http://retractionwatch.wordpress.com/ by two experienced science journalists picks up many--but not all--of the cases of peer-reviewed research papers being retracted later from science journals. Psychology as a discip…

The pressure on academics is indeed an issue. They have to teach, guide PhD's, research and publish (without mentioning controlling internal affairs).

On a side note. Has anyone found a good alternative for Mendeley? I heard http://bohr.launchrock.com/ is working on sth cool.

Re: Just 11% of 53 cancer research papers were reproducible

#79

Post by omnisci » Thu, May 09, 2013, 2:38 PM UTC

jpdoctor wrote: post #69

jballanc wrote: post #67

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being comp…

> It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.

After seeing how much of the research is done, I'd agree with this more.

This is a problem that is getting worse as funding is getting cut more, people feel they have a need to get a paper out, regardless of the results. You get positive results, make a story up about it, then run to publish it before trying to reproduce it or look further into the data. While this doesn't happen in every lab, I'm unhappy to say that I've seen this happen in many "high impact" labs.

Re: Just 11% of 53 cancer research papers were reproducible

#80

Post by jballanc » Thu, May 09, 2013, 2:41 PM UTC

omnisci wrote: post #79

jpdoctor wrote: post #69

Earlier quoted context omitted.

> It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.

After seeing how much of the research is done, I'd agree with this more. This is a problem that is getting worse as funding is getting cut more, people feel they have a need to get a paper out, regardless of the results. You get positive results, make a story up about it, then run to publish it before trying to reproduce it or look further into the data. While this doesn't happen in every lab, I'm unhappy to say that…

I've seen this happen as well, but I think the problem is more with the "make up a story" part and less with the "run to publish" part. I've seen really really interesting results that defy explanation get passed over for publication in favor of something more mundane that can "tell a story" because, it seems, stories get funded...intriguing research? not so much...