Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Just 11% of 53 cancer research papers were reproducible

nature.com

Post reply on HN
Print view View original

121–130 of 184 posts

Re: Just 11% of 53 cancer research papers were reproducible

#121

Post by spikels »

Retric wrote: post #77
jpdoctor wrote: post #69

Earlier quoted context omitted.

> It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.

If 11% of the papers are reproducible and people are trying to reproduce the results then your still making progress. Just the slow and expencive kind. Considering how complex the subject matter is I don's think it's reasonable to expect anything else. The problem is people are not trying to reproduce results which harms the field and slows everything down.

Actually very few people are trying to reproduce results. It is a lose-lose situation. Either you confirm the previous results which won't get published or you can't confirm them which means you are either not as competent as the original researcher or you have to embarrass your colleague and bring shame on your profession. Neither of which gets you published or helps you career or gets you more funding. The incentives are all screwed up.

These Amgen researcher had to sign agreements that they would not publish the results of their attempts to reproduce these experiments before they could get enough detail to attempt to reproduce them. Clearly this is not how science is supposed to work but it is exactly how businesses operate. Very sad.

Re: Just 11% of 53 cancer research papers were reproducible

#122

Post by epistasis »

spikels wrote: post #109
epistasis wrote: post #73

Earlier quoted context omitted.

You're assuming that only "reproducible" results are valid, which is incorrect. Results that are reproducible within a lab, but not when attempted by other people in different settings indicate that there's an unaccounted for variable, and that falls into the 89%. There's still something valuable there, it's just that we don't yet know all the variables. We could just give up, or we could try to continue study of som…

If its not reproducible it is not science.

Define reproducible, in the terms of the precise actions that people take. Is it reproducible if the same scientist repeats the experiment in the same lab and gets the same results? Because that's the current bar of reproducibility, and that 89% that is not "reproducible" certainly passed that bar.

What was being tested was a different lab, with different materials, trying to get the "same" results, for some definition of same. If you give 100 programmers an algorithms book, and tell them to produce code for a binary search, and only 25% of the programmers are able to make something that works, does that mean that binary search is only 25% reproducible?

If five different companies benchmark five different web frameworks for their application, and come to 2-4 different answers about which one is the 'best,' does that mean that the benchmarks are not reproducible? Of course not.

What's being highlighted here in this study is the extreme diversity of biological models. And one doesn't necessarily expect exact reproducibility in other people's hands, because we simply don't have technology to characterize every single aspect of a biological model, and it's impossible sometimes to even recreate the exact same biological context. Is something "reproducible" if it means that it replicates in 5% of other cell lines, 25% of other cell lines?

Re: Just 11% of 53 cancer research papers were reproducible

#123

Post by ars »

aidenn0 wrote: post #108
ars wrote: post #100

Earlier quoted context omitted.

How were those authors able to enforce this request? To be a proper paper it should not be necessary to contact the author in order to reproduce it. If you had to ask the author for more details then their paper was incomplete - perhaps that's why it could not be reproduced.

If you want to maximize your chances of reproducing the results, you will use identical apparatus and methods to what the original study used. That is often not possible to do just from reading the paper.

If the identical apparatus is essential to reproducing the result, then the description of that apparatus is part of the experiment. Omitting that means it's not a proper paper.

Re: Just 11% of 53 cancer research papers were reproducible

#124

Post by irollboozers »

pg wrote: post #106

I feel obliged to point out that these other journals are merely copying the real pioneer in this field: http://jir.com .

Negative results vary from field to field. In math or physics, a conclusive negative can be very important in deciding what to study next or disproving established theory. In biology, a negative usually just means "I haven't found what I'm looking for yet".

Re: Just 11% of 53 cancer research papers were reproducible

#125

Post by Avshalom »

spikels wrote: post #109
epistasis wrote: post #73

Earlier quoted context omitted.

You're assuming that only "reproducible" results are valid, which is incorrect. Results that are reproducible within a lab, but not when attempted by other people in different settings indicate that there's an unaccounted for variable, and that falls into the 89%. There's still something valuable there, it's just that we don't yet know all the variables. We could just give up, or we could try to continue study of som…

If its not reproducible it is not science.

Huge swaths of astronomy are functionally unreproducible. We can argue about the math but many phenomena exist as a single example and/or that's basically static on our time scales. The best we can do is see if the math seems to produce similar looking structures when (sparsely) simulated.

Re: Just 11% of 53 cancer research papers were reproducible

#126

Post by MrGunn »

spikels wrote: post #109
epistasis wrote: post #73

Earlier quoted context omitted.

You're assuming that only "reproducible" results are valid, which is incorrect. Results that are reproducible within a lab, but not when attempted by other people in different settings indicate that there's an unaccounted for variable, and that falls into the 89%. There's still something valuable there, it's just that we don't yet know all the variables. We could just give up, or we could try to continue study of som…

If its not reproducible it is not science.

It's really critical that we don't confuse research that's nor reproducible with fraud. Very few scientific theories survive unmodified over time, so lack of reproducibility isn't a criticism and we really need to move the debate past this. Every theory is expected to be inaccurate as it only explains the data using the understanding of the time, but this isn't an indictment of the research or the researcher and studies of outright fraud indicate that that actually only happens around 1% of the time.

Reproducibility isn't about calling out people whose work isn't reproducible, it's about identifying and promoting the most robust stuff.

Re: Just 11% of 53 cancer research papers were reproducible

#127

Post by spikels »

ars wrote: post #123
aidenn0 wrote: post #108

Earlier quoted context omitted.

If you want to maximize your chances of reproducing the results, you will use identical apparatus and methods to what the original study used. That is often not possible to do just from reading the paper.

If the identical apparatus is essential to reproducing the result, then the description of that apparatus is part of the experiment. Omitting that means it's not a proper paper.

If only it were so. Often important parts of the protocol exist only in a hand written entries in a lab notebook. Because very few people are reproducing experiments and it is lots of extra work and jornals have limited space all the required details are rarely included. This is a big part of the problem. The typical work around is to contact the original reaseach to fill in the missing details.

Re: Just 11% of 53 cancer research papers were reproducible

#128

Post by MrGunn »

epistasis wrote: post #122
spikels wrote: post #109

Earlier quoted context omitted.

If its not reproducible it is not science.

Define reproducible, in the terms of the precise actions that people take. Is it reproducible if the same scientist repeats the experiment in the same lab and gets the same results? Because that's the current bar of reproducibility, and that 89% that is not "reproducible" certainly passed that bar. What was being tested was a different lab, with different materials, trying to get the "same" results, for some definiti…

So there is a theory that if something doesn't reproduce it's because the other guy was just incompetent, and that may be the case, but just like everyone wants to believe they're above average, everyone will want to go to the theory that the other guys just aren't any good, when I suspect that that will be much less of a factor. At any rate, when you start trying these drugs on the wide diversity of the patient population, if they're not super robust, they don't be of much use anyways.

Re: Just 11% of 53 cancer research papers were reproducible

#129

Post by omnisci »

spikels wrote: post #113
omnisci wrote: post #79

Earlier quoted context omitted.

After seeing how much of the research is done, I'd agree with this more. This is a problem that is getting worse as funding is getting cut more, people feel they have a need to get a paper out, regardless of the results. You get positive results, make a story up about it, then run to publish it before trying to reproduce it or look further into the data. While this doesn't happen in every lab, I'm unhappy to say that…

Why do you say funding is getting cut? NIH's budget has almost doubled in the last decade [1] and many of the other funders have seen similar growth as well as new funders appearing every year. I don't think the problem is lack of funding but screwed up incentives. When medical reaseach became focused on funding the quality of the results suffered. And if the vast majority of landmark cancer research can't be reprodu…

http://news.sciencemag.org/scienceinsider/2013/05/nih-detail...

Already funded grants are getting cut ~20% across the board. There is a ton of cuts going on right now to the NIH budget, google it and take a look. This has been happening for years now, trying to get an RO1 (large research grant) is becoming more and more difficult and it isn't helped by the constant changing in requirements.

Every other point I agree 100% with, the culture change has to happen. Nothing is impossible, and it just takes the right people to make the right things to happen. Everyone recognizes the problem, I can't tell you how many times I hear people complaining about the same problems over and over. The problem is that they aren't taking action, and with no action, nothing is going to take place. While there are many efforts in place (my project being one of them, http://omnisci.org), they need to be implemented properly. The same rule of startups applies to science. The ideas/concept means nothing with proper execution.

While the culture change may be slow, the academic world is having a really hard time keeping up. NIH is also fighting to stay afloat. I have a few friends who work as program officers and they really have a negative outlook on the future of research funding.

Re: Just 11% of 53 cancer research papers were reproducible

#130

Post by omnisci »

irollboozers wrote: post #124
pg wrote: post #106

I feel obliged to point out that these other journals are merely copying the real pioneer in this field: http://jir.com .

Negative results vary from field to field. In math or physics, a conclusive negative can be very important in deciding what to study next or disproving established theory. In biology, a negative usually just means "I haven't found what I'm looking for yet".

And that is the exact problem with certain researchers. I just had to explain this to my undergrad who's project resulted in negative results (his 1'st study). I congratulated him, as he discovered that the treatment he gave did NOT effect the genes he looked into. He didn't understand, so I had to explain that his study (with the correct controls), provided information that is useful to science. Negative data = data. Data = good :)
Post reply on HN