Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Just 11% of 53 cancer research papers were reproducible

nature.com

Post reply on HN
Print view View original

81–90 of 184 posts

Re: Just 11% of 53 cancer research papers were reproducible

#81

Post by omnisci »

atrus wrote: post #48
gordaco wrote: post #23

Wow, that's sad. Are we really so blind when it comes to cancer research? I recall that oncology journals usually have a ludicrously high impact index (as ludicrously as 5 digit IIRC); that means there are a lot of citations, which is an indicator that there is a lot of research going on. And with a lot of research going on, well, you can expect a lot of false positives. So, I'm wondering, could this be a case of che…

But who is going to pay to reproduce that research? What if that research took years to do?

You don't have to do the entire experiment, you can just do a proof of principle and then build on it. For example, a lot of studies in science have 1 big idea, and they show it works using a variety of different methods. Pick 1 method, show consistencies, and then build on it.

One of the serious problems in science is that people state what you said above and will jump in feet first on a huge multi thousand $ study, but never spend the time to validate the model in their own lab. So you waste that money initially, then go back to figure out what went wrong. $$$$ being wasted. (The lack of negative data publications is also a massive problem that contributes to this)

What I'd like to see (and I'm currently working on) is raw data from labs. I do xyz experiment, publish it, and release all the raw images, excel files, raw data outputs etc. Now other scientists can go into those data, ask their questions and try to reproduce it or at lease provide a different perspective. This is what I'm working on at http://omnisci.org and I intend to bring to the scientific community. We need transparency, because this "behind the door" shit (peer review, grant reviews, only positive data etc) isn't working.

Re: Just 11% of 53 cancer research papers were reproducible

#82

Post by vixen99 »

kailuowang wrote: post #18
viggity wrote: post #7

After hearing similar things about psychology papers, this is rather disconcerting. This is why I am a climate change skeptic, I don't know whether it is happening due to CO2 or not, but I am confident that science can't be confident when they can't do control experiments. You can't control for any variable when it comes to the climate, let alone all the reasonable ones. We have infinitely more capabilities to contro…

The possibility of CO2 causing global warming isn't 100% but it isn't very low either. It is the risk we are talking about, the risk of doing nothing and let the man made green house (if there is one) turn earth into an irreversible disastrous environment. Most CO2 emitting energy sources are not sustainable anyway and many of them emit other proved pollution as well. There is really nothing lost to going green energ…

Without qualification, I'm afraid what you're saying is parlously close to mere verbiage. Even the most ardent climate sceptic does not object to solar, wind or water energy under certain conditions. What is 'Going Green' then? In the UK it currently means, for instance, paying wind farms a million pounds sterling a day to produce no energy at all. Meanwhile poor people die because they can't afford to keep themselves warm thanks to horrendous energy bills punped up by huge subsidies to the likes of windmill owners and owners of land, hosting windmills.

Re: Just 11% of 53 cancer research papers were reproducible

#83

Post by campnic »

epistasis wrote: post #45
campnic wrote: post #27

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch t…

This is an entirely different issue than code; code mostly does the same thing when you run it twice. There's no such guarantee in biology. A cancer cell line growing in one lab may behave differently than descendants of those cells in a different lab. This may be due to slight differences in the timings between feeding the cells and the experiments, stochastic responses built into the biology, slight variations betw…

Oh, I agree. Biological experiment reproducibility is an incredibly hard problem. You are probably right that it is 'trivial' by comparison in the same way that landing on mars is trivial to landing on Alpha Centauri.

Re: Just 11% of 53 cancer research papers were reproducible

#84

Post by campnic »

phireal wrote: post #34
campnic wrote: post #27

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch t…

I think that's what the folks at Software Carpentry [0] are trying to do. I went on one of their courses, and you're taught the basics of writing good software, version control and databases (SQLite). I've frequently recommended it to fellow scientists. [0] http://software-carpentry.org/

This is great! Thanks for sharing.

Re: Just 11% of 53 cancer research papers were reproducible

#85

Post by japhyr »

This is one of the reasons I am happy to see what is going on with the Center for Open Science [0]. The goal of the project is to open up the entire research process, not just create open access to published results. They aim to make open tools supporting the entire research process as well, through the Open Science Framework [1].

One of the benefits of this approach is to provide peer review at each step of the research process. There is also an emphasis on encouraging validation of prior results, rather than having everyone focus on creating new research. This is the focus of the Reproducibility Project [2].

The Center is just getting started, and I sure hope it takes off.

[0] - http://centerforopenscience.org/

[1] - http://openscienceframework.org/

[2] - http://openscienceframework.org/project/EZcUj/wiki/home

Re: Just 11% of 53 cancer research papers were reproducible

#86

Post by brandon_wirtz »

This has been going around along with the Chemo Therapy doesn't work thread.

The problem with people like the author is they think all cancer is the same. They try to apply information about Hodgkins to non-hodgkins. They think "Breast cancer" is a disease, not a symptom. There over 40 different cancer causes for breast cancer. And the treatments that work are primarily based on the cause not the visible symptom.

This makes studies hard. Most people never learn the cause of their cancer. We are not fortunate enough to always have a single known cause that says "yes you worked in a nuclear waste plant for 6 years" and know that was the cause.

Chemo for example in Hodgkin's Lymphoma increase your 5 year survival chances by 45%. But if you have Smoking related Lung Cancer it is less than 2% difference.

Nature.com rarely produces articles that are informed. They push an agenda of Holistic medicine at the expense of scientific research. I am all for non-traditional medicine, but I don't discount the advances from University and Clinical research.

Re: Just 11% of 53 cancer research papers were reproducible

#87

Post by [deleted] »

campnic wrote: post #83
epistasis wrote: post #45

Earlier quoted context omitted.

This is an entirely different issue than code; code mostly does the same thing when you run it twice. There's no such guarantee in biology. A cancer cell line growing in one lab may behave differently than descendants of those cells in a different lab. This may be due to slight differences in the timings between feeding the cells and the experiments, stochastic responses built into the biology, slight variations betw…

Oh, I agree. Biological experiment reproducibility is an incredibly hard problem. You are probably right that it is 'trivial' by comparison in the same way that landing on mars is trivial to landing on Alpha Centauri.

[deleted]

Re: Just 11% of 53 cancer research papers were reproducible

#88

Post by jballanc »

kirk21 wrote: post #78
tokenadult wrote: post #57

This editorial commentary and the article on which it is based are part of an ongoing effort to improve the quality of scientific publication in a number of disciplines. The Retraction Watch group blog http://retractionwatch.wordpress.com/ by two experienced science journalists picks up many--but not all--of the cases of peer-reviewed research papers being retracted later from science journals. Psychology as a discip…

The pressure on academics is indeed an issue. They have to teach, guide PhD's, research and publish (without mentioning controlling internal affairs). On a side note. Has anyone found a good alternative for Mendeley? I heard http://bohr.launchrock.com/ is working on sth cool.

Have you considered Papers? http://papersapp.com/

Re: Just 11% of 53 cancer research papers were reproducible

#90

Post by cowsandmilk »

Thrymr wrote: post #72
campnic wrote: post #27

I have been in discussions about this with one of my friends working in academic materials research. Its amazing the amount of work today done by scientist at universities writing code without very basic software development tools. I'm talking opening their code in notepad, 'versioning' files by sending around zip files with numbers manually added to the end of the file name, etc. This doesn't even begin to scratch t…

Recent article on git and reproducability in science: http://www.scfbm.org/content/8/1/7 It is badly needed.

That article says "Data are ideal for managing with Git."

I one time tried using git to manage my data. The problem is, I frequently have thousands of files and gigabytes of data. And git just does not handle that well.[1]

One time, I even tried building a git repo that just had the history of pdb snapshots. The PDB frequently has updates, and I have run into many cases where an analysis of a structure was done in a paper 3 years ago, but the structure has been updated and changed since then, making the paper make no sense until I thought to look at the history of changes to the structure. Unfortunately, git could not handle this at all when I tried it, taking days to construct the repo and then that repo was unbearably slow when I tried to use it.

Git would probably work well for storing the data used by most bench scientists, but for a computational chemist puking up gigabytes of data weekly on a single project, it is sadly horrible for handling the history of your data.

[1] http://osdir.com/ml/git/2009-05/msg00051.html

Post reply on HN