Live data from Hacker News

Just 11% of 53 cancer research papers were reproducible

nature.com

61–70 of 184 posts

Re: Just 11% of 53 cancer research papers were reproducible

#61
post #4

This is one of the reasons I went into math and software. In math there is only black and white, provably true and patently wrong. Still, part of the reason this is so is that to show any results everything must be laid on the table. CS and Math papers are therefore open by design and the world has benefited. I hope the rest of science will move in this direction.

I'm guessing you have not read Proofs and Refutations by Imre Lakatos. It is worthwhile.

Also if you read on the history of mathematics, the various back and forth foundational issues raised by infinitesmals, Fourier series, proof by contradiction and the like are very interesting. Sure, it is possible for a modern mathematician to look at all that and say, They just didn't know how to do it, HERE you go. But over decades and centuries people were somehow successfully continuing to do math, even though they knew that something serious was wrong in their understanding.

Re: Just 11% of 53 cancer research papers were reproducible

#62
post #51

Perhaps we are looking for the wrong measure of success. This is not inclined plane ball rolling in fifth period Physics with Mr. Johannes. Reproducibility in cutting edge experiment is success. It means that what is being measured - even if the wrong thing - is within the control of the experimenter. It is absurd, but practical, to publish experimental results with the implication that they are reproducible based on…

I wonder if it would be possible to produce a journal that only released an issue when it has, say, 10 articles? No monthly/quarterly schedule, you could have an issue release next week, or next year.

As a digital document it is easy. Because printing requires substantial coordination, it is less practical. The subscription model common to the scientific journal industry makes it contractually problematic.

I wonder what portion of academic research is equivalent to content farming.

Re: Just 11% of 53 cancer research papers were reproducible

#63
post #59

I work in biomedical research, and this finding has been discussed quite broadly. Most researchers don't believe it. Exactly reproducing novel findings usually requires a significant investment into the underlying procedures, which most places will not undertake. Reproduction instead usually occurs as part of an extension of the initial findings. The complexity of biology means that the first findings are frequently…

Really?

Meaning that either there's another unaccounted variable(s) that controls the effect of A in B and C, meaning that we cannot conclude anything from the experiment.

Yes and no is not acceptable...

Re: Just 11% of 53 cancer research papers were reproducible

#64
post #60
post #54

Earlier quoted context omitted.

In practical terms, reproducing someone else's work most of the time boils down to redoing someone else's PhD thesis. Which is both not very interesting and doesn't help getting your own research done.

I agree that reproducing someone else's experiment won't help you get your PhD closer to completion (because you should be doing your own experiments and publishing no matter what, if you expect to land that lecturing position, i.e. publish or perish). Reproducibility is one of the main principles of the scientific method (those papers shouldn't even been accepted if they don't contain enough information about how to…

Yes, I'm not arguing the foundations of the scientific method, just pointing out how it is in practice.

In every field you have the established base theory, the bleeding edge you work on, and a bunch of preliminary work in-between. Checking out references is important, but this is a recurrent process (as you need to check the references of those works too), and you only have so much time.

So what really seems to happen is only a few select works in any subfield achieve large citation index, and those stand a decent chance of being verified at some point, or at least they do have enough work trying to build up on their results that a systematic inconsistency would show up.

Re: Just 11% of 53 cancer research papers were reproducible

#65
post #42

Earlier quoted context omitted.

Agreed, but this study was specifically designed to highlight the problem not solve it. And the results would be useless, or non-existent, if they restricted themselves to scientists that would co-operate fully with the process and the possible "shaming" afterward.

I think it also highlights the underlying mentality that prestige matters more than integrity and "if everyone does it then it's fine". If tax-funded scientists can't uphold high standards for themselves, the public is entitled to demand them to do so.

You might be being a little harsh. There is a huge difference between "hey, I'm having a problem reproducing your results, want to help me either reproduce or disprove your results?", which legitimate scientists will and do participate in, and "I'm doing a study on unreproducible results and want to publish your paper and name in the list afterwards".

Notice they say they got help without the shaming part so the scientists were willing to participate, just not get listed.

I personally know scientists who have helped out with disproving their own findings, it's not uncommon.

Re: Just 11% of 53 cancer research papers were reproducible

#66
post #49

What is more troubling is not that so few of these results results are reproducible, but that it appears almost no-one is trying to reproduce the results of earlier studies. The ability of the scientists who wrote the paper to even get access to the resources necessary to try and reproduce the results is limited. I'm reminded of Feymann's Cargo Cult Science: "I was shocked to hear of an experiment done at the big acc…

If I had to sum up science in one phrase, I wouldn't say anything about the "scientific process" or anything like that. I would say: "Look for reasons why you are wrong, not reasons why you are right." You can always find reasons why you're right. You can always do an astrology forecast and find someone for whom it was dead-on accurate. That's not the problem with astrology, the problem lies in how often it is wrong. It's wrong so often it's useless. But if you only examine the positive evidence in favor of it, you will never come to that conclusion.

The theories that are powerful and worthwhile are the ones that are rarely or never wrong. Can't always get "never". It's a complicated world and we aren't all physicists. But we at least ought to be able to get "rarely", and if you can't, well, I guess it's not science then. That's OK. Unfortunately, not everything is amenable to science, though you can still approach it in this spirit of trying to see how you might be wrong rather than proving yourself right.

Once you start looking around with that standard, it's not hard to see how little science is really being done. Why are we publishing these dubious studies? Because for all the scientific trappings we claim, with statistics and p-values and carefully-written recordings of their putative procedure written in precisely the right way to make it sound like everything was recorded (while still leaving out an arbitrary number of relevant details), we've created a system where we are telling people to look for reasons why they are right... or we won't publish their results. Guess what kind of results we get with that?

If you start from the idea that you need to look for why you are wrong, the scientific method will fall out of that, along with any local adjustments and elaborations you may need, and every discipline, sub-discipline, and indeed at times even individual experiments need adjustments. If you start with "The Scientific Method", but you don't understand where it came from, how to use it, or what it is really telling you, you'll never get true science, just... noise.

Re: Just 11% of 53 cancer research papers were reproducible

#67
I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on...

Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being completely worthless, snake-oil type stuff, but this was not the case at all. Rather, individuals were working on an area about which almost nothing was known, and (more importantly) for which the key central organizing laws were not yet revealed (for Alchemy: the atomic theory of matter and chemical bonding, for Cell Signaling: how individual proteins and small molecules interact). Furthermore, the goals were lofty and almost certainly unattainable (for Alchemy: turn base metals into gold, for Cell Signaling: cure cancer), driving people to do "rush" work. It's not that the results are completely invalid, or that the experiments are useless. It's just that everyone feels like they're so close to a solution (Alchemists were way off with Phlogiston, and I'll bet Cell Signaling researchers are similarly clueless as to what really matters) that no one takes the time to step back and synthesize the results in an attempt to understand the forest from the trees.

Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev.

From experience, the fact that 89% of these "cancer research" papers are not reproducible almost definitely has less to do with fraudulent data and so much more to do with crazy complex experimental setups that end up probing a half-dozen experimental variables all at once (without the researchers even grasping that this is going on).

Yeah, publish or perish sucks, but what sucks more is the death of basic science research. The Alchemists eventually became Chemists because they refocused on core principles (atomic theory, bonding) and forgot about the lofty goals (turn lead into gold)...

...but try telling any politician that they should decrease cancer research funding and refocus on genetics, structural biology, and evolution research. I'd love to know how they respond.

Re: Just 11% of 53 cancer research papers were reproducible

#68
The most interesting thing about all this is the fact that the researchers were required by paper authors to sign non-disclosure agreements and agree not to identify papers with non-reproducible results. This says a lot about the confidence authors have in their papers, doesn't it?

Sorry but this isn't science anymore.

Re: Just 11% of 53 cancer research papers were reproducible

#69

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being comp…

> It's not that the results are completely invalid

Especially if only 11% of them are reproducible, they're only 89% invalid.

I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.

Re: Just 11% of 53 cancer research papers were reproducible

#70
post #63
post #59

I work in biomedical research, and this finding has been discussed quite broadly. Most researchers don't believe it. Exactly reproducing novel findings usually requires a significant investment into the underlying procedures, which most places will not undertake. Reproduction instead usually occurs as part of an extension of the initial findings. The complexity of biology means that the first findings are frequently…

Really? Meaning that either there's another unaccounted variable(s) that controls the effect of A in B and C, meaning that we cannot conclude anything from the experiment. Yes and no is not acceptable...

There's 20,000 broad classes of variables (genes), and on average 12 variations of each of those variable (alternative splicing, post-translational modification, cellular localization, other variables we haven't yet discovered ). This series of two experiments and likely 2 years found associations between three proteins, which is a hook into system of great complexity, a way to start looking for more.

It is not "reproducible" yet it is incredibly valuable.

Post reply on HN