Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Just 11% of 53 cancer research papers were reproducible

nature.com

Post reply on HN
Print view View original

111–120 of 184 posts

Re: Just 11% of 53 cancer research papers were reproducible

#111

Post by Brashman »

rdfi wrote: post #63
pcrh wrote: post #59

I work in biomedical research, and this finding has been discussed quite broadly. Most researchers don't believe it. Exactly reproducing novel findings usually requires a significant investment into the underlying procedures, which most places will not undertake. Reproduction instead usually occurs as part of an extension of the initial findings. The complexity of biology means that the first findings are frequently…

Really? Meaning that either there's another unaccounted variable(s) that controls the effect of A in B and C, meaning that we cannot conclude anything from the experiment. Yes and no is not acceptable...

Yes and no is acceptable. It's really a "Yes, but...". Is Newtonian physics correct? Yes and no. It's roughly correct at a macroscopic level, but at small scales quantum effects are important and at high speeds relativistic effects are important. "It's correct, but..."

Not every work of science needs to be exact fact before being published. Really, nothing in science is accepted as absolute fact. The scientific publishing process is a conversation between scientists to try to determine the truth.

Re: Just 11% of 53 cancer research papers were reproducible

#112

Post by jurassic »

campnic wrote: post #27

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch t…

> Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools.

I agree 100%. I recently quit my PhD so I still know a lot of people on the frontlines of science. One of these friends recently asked me to help them with a coding issue so they gave me an ssh login to group's server. I login and start reading the source.

It was all Fortran, with comments throughout like "C A major bug was present in all versions of this program dated prior to 1993." What bug, and of what significance for past results? Unknowable. As far as I can tell from the comments, the software has been hacked on intermittently by various people of various skill since at least 1985 without ever using source control or even starting a basic CHANGELOG describing the program's evolution. The README is a copy/paste of some old emails about the project. There are no tests.

So even though computer modeling projects should, in theory, be highly reproducible... it often seems like researchers are not taking the necessary steps to know what state their codebase was in at the time certain results were obtained.

Re: Just 11% of 53 cancer research papers were reproducible

#113

Post by spikels »

omnisci wrote: post #79
jpdoctor wrote: post #69

Earlier quoted context omitted.

> It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.

After seeing how much of the research is done, I'd agree with this more. This is a problem that is getting worse as funding is getting cut more, people feel they have a need to get a paper out, regardless of the results. You get positive results, make a story up about it, then run to publish it before trying to reproduce it or look further into the data. While this doesn't happen in every lab, I'm unhappy to say that…

Why do you say funding is getting cut? NIH's budget has almost doubled in the last decade [1] and many of the other funders have seen similar growth as well as new funders appearing every year.

I don't think the problem is lack of funding but screwed up incentives. When medical reaseach became focused on funding the quality of the results suffered. And if the vast majority of landmark cancer research can't be reproduced much of that money was wasted.

The solution will require a huge cultural change which may be impossible. However step one is recognizing the problem. And some efforts are already underway such as journals like PLoS which publish negative results and more recently The Reproducibility Project and Reproducibility Initiative [2,3]. Still it will be difficult.

[1] http://officeofbudget.od.nih.gov/pdfs/spending_history/Mecha...

[2] http://www.openscienceframework.org/project/EZcUj/wiki/home

[3]https://www.scienceexchange.com/reproducibility

Re: Just 11% of 53 cancer research papers were reproducible

#114

Post by spikels »

tokenadult wrote: post #57

This editorial commentary and the article on which it is based are part of an ongoing effort to improve the quality of scientific publication in a number of disciplines. The Retraction Watch group blog http://retractionwatch.wordpress.com/ by two experienced science journalists picks up many--but not all--of the cases of peer-reviewed research papers being retracted later from science journals. Psychology as a discip…

I would like to add The Reproducibility Project and Reproducibility Initiative to your list. They are seeking to specifically address this issue.

http://www.openscienceframework.org/project/EZcUj/wiki/home https://www.scienceexchange.com/reproducibility

Re: Just 11% of 53 cancer research papers were reproducible

#115

Post by bargl »

spikels wrote: post #109
epistasis wrote: post #73

Earlier quoted context omitted.

You're assuming that only "reproducible" results are valid, which is incorrect. Results that are reproducible within a lab, but not when attempted by other people in different settings indicate that there's an unaccounted for variable, and that falls into the 89%. There's still something valuable there, it's just that we don't yet know all the variables. We could just give up, or we could try to continue study of som…

If its not reproducible it is not science.

https://news.ycombinator.com/item?id=5680310

Re: Just 11% of 53 cancer research papers were reproducible

#116

Post by elizabethiorns »

We are attempting to solve this by identifying and rewarding reproducible research (www.reproducibilityinitiative.org). However, it has been incredibly difficult to get funding to conduct the validation studies due to the obsession of funding "novel" results, rather than funding the replication studies required to identify reproducible research. Funders need to step up to solve this problem.

Re: Just 11% of 53 cancer research papers were reproducible

#117

Post by spikels »

api wrote: post #105
evoloution wrote: post #41

Earlier quoted context omitted.

Such a powerful sentence. Lately science looks like olympic games without anti-doping control...

I'm becoming pretty convinced that the human future depends on the disruption and obsolescence of bureaucracies.

I think you are on to something. When science became so big that it first needed then eventually was run by "managers" I bet it lost much of its effectiveness. But how do we change this? For starters there is a lot of money involved for the institutions conducting this (apparently shoddy) research.

Re: Just 11% of 53 cancer research papers were reproducible

#118

Post by kanzure »

spikels wrote: post #117
api wrote: post #105

Earlier quoted context omitted.

I'm becoming pretty convinced that the human future depends on the disruption and obsolescence of bureaucracies.

I think you are on to something. When science became so big that it first needed then eventually was run by "managers" I bet it lost much of its effectiveness. But how do we change this? For starters there is a lot of money involved for the institutions conducting this (apparently shoddy) research.

> When science became so big that it first needed then eventually was run by "managers" I bet it lost much of its effectiveness.

You can just do projects on your own, you know. There's nothing about science that absolutely requires an institution.

http://groups.google.com/group/diybio

Post reply on HN