Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Just 11% of 53 cancer research papers were reproducible

nature.com

Post reply on HN
Print view View original

31–40 of 184 posts

Re: Just 11% of 53 cancer research papers were reproducible

#31

Post by jacobparker »

gordaco wrote: post #15
rajeevk wrote: post #6

Earlier quoted context omitted.

CS has similar problem, specially in AI fields. Researches often claim the accuracy rate more than 95%, but in reality that is much lesser (in fact less than 10% in most of the claims).

Having worked in computer vision and robotics, I can attest to this.

Yes. Of the fields I've implemented algorithms from papers in, computer vision seems to be particularly adverse to discussing the (often critical) downsides of their algorithms. Often its something like "Camera calibration isn't exactly perfect? Well this scene reconstruction technique simply wont work."

Re: Just 11% of 53 cancer research papers were reproducible

#32

Post by pesenti »

The problem is not that research is not reproducible. Researchers are humans and full of biases. The problem is too much emphasis is put on novelty vs. reproducibility. It took a long time for software engineers to recognize the value of testing/validating vs. creating new code. In the same way the research world needs to put new emphasis on reproducing/validating results.

Re: Just 11% of 53 cancer research papers were reproducible

#33

Post by drharris »

ndr wrote: post #28
drharris wrote: post #14

Earlier quoted context omitted.

Um, 6 papers out of 53, significant digits give you 11%. Not that difficult.

I guess saying "11% of 53 papers" was cooler than saying "6 out of 53". Meh.

Yeah, my guess is they used that form to imply 11% of all cancer research is unreproducible. But in ironic fashion, they can't show their own data and thus we can make no broad claims about it.

Re: Just 11% of 53 cancer research papers were reproducible

#34

Post by phireal »

campnic wrote: post #27

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch t…

I think that's what the folks at Software Carpentry [0] are trying to do. I went on one of their courses, and you're taught the basics of writing good software, version control and databases (SQLite). I've frequently recommended it to fellow scientists.

[0] http://software-carpentry.org/

Re: Just 11% of 53 cancer research papers were reproducible

#35

Post by baldfat »

Pediatric Cancer gets only 4% of Federal funding and less than 1% of Private Drug company research. Also American Cancer Society less than 5 cents on the dollar goes to research.

Pediatric Cancer is the leading cause of death for 15 and under.

So to see this funding for cancer get thrown away by fraud is extremely upsetting.

Re: Just 11% of 53 cancer research papers were reproducible

#36

Post by return0 »

pesenti wrote: post #32

The problem is not that research is not reproducible. Researchers are humans and full of biases. The problem is too much emphasis is put on novelty vs. reproducibility. It took a long time for software engineers to recognize the value of testing/validating vs. creating new code. In the same way the research world needs to put new emphasis on reproducing/validating results.

The problem is that too much emphasis is put on prestige and reputation than on actual impact of the work. Coming from a physics background I always felt that life scientists tend to be overly audacious in their claims, am I alone in this?

Re: Just 11% of 53 cancer research papers were reproducible

#37

Post by Ensorceled »

return0 wrote: post #30
Ensorceled wrote: post #25

Earlier quoted context omitted.

Nature wrote an editorial justifying the situation, you can read it here http://www.nature.com/nature/journal/v485/n7396/full/485041e...

> "those authors required them to sign an agreement that they would not disclose their findings about specific papers" I don't know why this doesn't cause a huge stir in the scientific community. Seems like everyone is fine with people sweeping their negative data under the rug. Shameful is a very mild word for that.

Agreed, but this study was specifically designed to highlight the problem not solve it. And the results would be useless, or non-existent, if they restricted themselves to scientists that would co-operate fully with the process and the possible "shaming" afterward.

Re: Just 11% of 53 cancer research papers were reproducible

#38

Post by bigiain »

campnic wrote: post #27

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch t…

Have you seen: http://matt.might.net/articles/crapl/

"Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it into running; and it's not pretty code. Academic code is about "proof of concept." These rough edges make academics reluctant to release their software. But, that doesn't mean they shouldn't.

Most open source licenses (1) require source and modifications to be shared with binaries, and (2) absolve authors of legal liability.

An open source license for academics has additional needs: (1) it should require that source and modifications used to validate scientific claims be released with those claims; and (2) more importantly, it should absolve authors of shame, embarrassment and ridicule for ugly code."

Re: Just 11% of 53 cancer research papers were reproducible

#39

Post by brudgers »

Perhaps we are looking for the wrong measure of success. This is not inclined plane ball rolling in fifth period Physics with Mr. Johannes. Reproducibility in cutting edge experiment is success. It means that what is being measured - even if the wrong thing - is within the control of the experimenter.

It is absurd, but practical, to publish experimental results with the implication that they are reproducible based on a peer review of the results. "Well they look reproducible to me," when uttered by a peer reviewer is the current standard. All we are seeing is that the a priori reasoning of experts is not a substitute for empirical investigation of scientific claims.

Let us not forget that a journal which finds no worthy articles this quarter has only two options, publish unworthy ones or publish no articles. I know how I react when a magazine subscription does not come.

Re: Just 11% of 53 cancer research papers were reproducible

#40

Post by Anderkent »

gordaco wrote: post #23

Wow, that's sad. Are we really so blind when it comes to cancer research? I recall that oncology journals usually have a ludicrously high impact index (as ludicrously as 5 digit IIRC); that means there are a lot of citations, which is an indicator that there is a lot of research going on. And with a lot of research going on, well, you can expect a lot of false positives. So, I'm wondering, could this be a case of che…

Medical research in general is dubious at best. See for example http://www.plosmedicine.org/article/info:doi/10.1371/journal...

There's even a startup trying to work around it: http://www.metamed.com/. They look for papers related for a given condition on request, trying to find ones that are actually well done and promising.

Post reply on HN