It seems like a big win for science could be had by centralizing some aspects of research within academic and research institutions. There should be well-funded software czars and statistics czars at each university that facilitate the efforts of the individual researchers to try to reduce the amount of shoddy science that gets published. Researchers are really terrible at this, and presumably slower than dedicated people even who are working on many different projects, so the division of labor should make the whole apparatus vastly more efficient, able to write and win more grants, etc. Of course, many researchers would fight against this because if you can't publish shoddy research, you can't publish as much research, but I would think there should be some way to get there from here.
Just 11% of 53 cancer research papers were reproducible
101–110 of 184 posts
Re: Just 11% of 53 cancer research papers were reproducible
#102This is one of the reasons I went into math and software. In math there is only black and white, provably true and patently wrong. Still, part of the reason this is so is that to show any results everything must be laid on the table. CS and Math papers are therefore open by design and the world has benefited. I hope the rest of science will move in this direction.
If that were an actual possibility in other fields everyone would be delighted. Sadly, biology (human minds and societies as well) are much too complicated for this to work.
This is an actually challenging problem. I don’t think running away from it and resorting to cheap tribalism is a great idea.
Re: Just 11% of 53 cancer research papers were reproducible
#103I read a lot of medical research papers and I'd say about 1 in 10 is about cancer. They usually go something like this: We found this receptor on a cancer cell and it caused it to grow or shrink. We therefore propose making a drug to inhibit or act as an agonist for this receptor site which also affects an enormous amounts of completely un-releated stuff around the body in order to fight cancer.
Truth is, they may have just gotten a funny little mutant of a cancer cell that happens to heavily express this one receptor as part of its unique mutated biology. For all we know a lot of cancer research has been about exploring the peculiarities of the genome of Henrietta Lacks.
Re: Just 11% of 53 cancer research papers were reproducible
#104I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being comp…
>Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev. That's very optimistic, but not necessarily true (even ignoring the question of when ). Priestley succeeded so well because he developed simple, reproducible experiments (expose mercury oxide to sunlight to produce oxygen etc.) that he was happy to share for no financial gain (even traveling to others' labs to help them reproduce his w…
I don't understand why science should be a closed guild in the first place.
Re: Just 11% of 53 cancer research papers were reproducible
#105After decades of publish-or-perish sweatshop science I'm sure the great shining archive of scientific truth is sort of like an inbox with no spam filter.
Such a powerful sentence. Lately science looks like olympic games without anti-doping control...
I'm becoming pretty convinced that the human future depends on the disruption and obsolescence of bureaucracies.
Re: Just 11% of 53 cancer research papers were reproducible
#106I feel obliged to point out that these other journals are merely copying the real pioneer in this field: http://jir.com.
Re: Just 11% of 53 cancer research papers were reproducible
#107I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on... Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being comp…
>Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev. That's very optimistic, but not necessarily true (even ignoring the question of when ). Priestley succeeded so well because he developed simple, reproducible experiments (expose mercury oxide to sunlight to produce oxygen etc.) that he was happy to share for no financial gain (even traveling to others' labs to help them reproduce his w…
Part of why it may not be true is that we lack the language or conceptual vocabulary to describe how living systems really work.
When we engineer things, it's one-part-one-function. Living systems are every-part-every-function network graphs with weighted edges that are subject to dynamic reconfiguration.
A cell isn't a computer that runs "code" in the form of DNA, nor is it a "machine" as we understand it. A "structured cloud of probabilistic quantum interactions among molecular nanomachines" is a bit more accurate.
Until we conquer these meta-problems, we won't understand the cell or the genome. I don't think they can be understood classically -- and I don't mean classically in the sense of omitting quantum mechanics. (Though that's true too.) I mean classically in the sense of linear fixed-relationship cause and effect machines.
Re: Just 11% of 53 cancer research papers were reproducible
#108Earlier quoted context omitted.
> "those authors required them to sign an agreement that they would not disclose their findings about specific papers" I don't know why this doesn't cause a huge stir in the scientific community. Seems like everyone is fine with people sweeping their negative data under the rug. Shameful is a very mild word for that.
How were those authors able to enforce this request? To be a proper paper it should not be necessary to contact the author in order to reproduce it. If you had to ask the author for more details then their paper was incomplete - perhaps that's why it could not be reproduced.
If you want to maximize your chances of reproducing the results, you will use identical apparatus and methods to what the original study used. That is often not possible to do just from reading the paper.
Re: Just 11% of 53 cancer research papers were reproducible
#109Earlier quoted context omitted.
> It's not that the results are completely invalid Especially if only 11% of them are reproducible, they're only 89% invalid. I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.
You're assuming that only "reproducible" results are valid, which is incorrect. Results that are reproducible within a lab, but not when attempted by other people in different settings indicate that there's an unaccounted for variable, and that falls into the 89%. There's still something valuable there, it's just that we don't yet know all the variables. We could just give up, or we could try to continue study of som…
If its not reproducible it is not science.
Re: Just 11% of 53 cancer research papers were reproducible
#110I work in biomedical research, and this finding has been discussed quite broadly. Most researchers don't believe it. Exactly reproducing novel findings usually requires a significant investment into the underlying procedures, which most places will not undertake. Reproduction instead usually occurs as part of an extension of the initial findings. The complexity of biology means that the first findings are frequently…
Really? Meaning that either there's another unaccounted variable(s) that controls the effect of A in B and C, meaning that we cannot conclude anything from the experiment. Yes and no is not acceptable...
You're wrong. What you say - that we conclude anything if not everything was taken into consideration - is undoubtedly true for mathematical meaning of "conclude". Mathematicians, computer scientists and a few others, like theoretic physicists, have a luxury of using law of excluded middle, which Sherlock Holmes also used. The famous detective states that if you eliminate everything that is impossible, whatever remains is true, even if it sounds improbable.
All the people who use this law do so because their work is about some kind of model, which can be wholly known. However, if you have a misfortune of working in the real world - like nearly everyone - then you can't apply this law. This means that you won't ever get a proof in a mathematical sense. You won't ever be completely certain - you can be convinced beyond reasonable doubt, but that's all.
So when we talk about how biologists are just grant hunters because someone couldn't reproduce their experiments we need to take this into account. I don't know, but if I had to guess I'd say that nobody ever expected these experiments to be 100% accurate, 100% reproducible or 100% true. I think they are treated as a data point, some input to think of, and not definite truth.
But I may be completely wrong here, of course.