Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why most published scientific research is probably false [video]

economist.com

Post reply on HN
Print view View original

41–50 of 57 posts

Re: Why most published scientific research is probably false [video]

#41

Post by epistasis »

capnrefsmmat wrote: post #37
epistasis wrote: post #21

Earlier quoted context omitted.

In the important areas, it does check itself. If a scientist has been doing some interesting work, she will soon find herself with colleagues working on the same area. If she has not been doing something interesting, that line of research does not get cited, dies out, and does not get funding. In active learning, there's a tradeoff between exploration and exploitation. Science does this too, but heuristically.

Not necessarily true. Ioannidis did a review of 49 studies cited more than 1,000 times in the medical literature -- prime candidates for being replicated or tested by future results. Of those, - 16% were found by later studies to be wrong - 16% were found to have exaggerated the size of effects they claimed to detect - 44% were replicated, and - 24% were unchallenged and unreplicated by later literature. So a full qu…

Thank you for a very informative and helpful reply! However, I would like to see this work validated before I fully accept his conclusions.

Regarding the 24%, if you look in Table 2 of that study, of the 12 that were "unchallenged and unreplicated", the notes for 10 list follow on studies that would reveal problems with the initial study, were the initial study to be "wrong."

That leaves 4% that are "unchallenged and unreplicated" yet have 1000 citations. So I would be quite interested to know the take of the researchers in the field on those papers, and if they would agree with Ioannidis' assessment on those 2 papers.

Re: Why most published scientific research is probably false [video]

#42

Post by jamesaguilar »

mjn wrote: post #7
jamesaguilar wrote: post #2

Probably false? As in you have a better chance claiming the negation of a scientific paper's conclusion than the actual conclusion? I doubt it.

In certain fields, that's not implausible. Many results are along the lines of "compound X is effective against disease Y", where the negation, "compound X is not effective against disease Y", is a reasonable baseline assumption, because most compounds are not effective against most diseases. Results where the prior odds could plausibly be considered 50/50 are another story. Scientific research that doesn't take the…

That's a good point. I didn't consider the fact that possibly many discoveries have a very low prior expectation of truthood.

Re: Why most published scientific research is probably false [video]

#43

Post by snowwrestler »

It is a mistake of reasoning to take a meta-analysis of medical research and expand its conclusions to the rest of science.

Medical research has a number of peculiarities among the sciences, including the complexity of its subject (perhaps the highest of any discipline), the emotional reaction to the subject, the speed at which people try to turn scientific findings into products or advice, and the concommittal eagerness to trust epidemiological results without a known physical mechanism.

It's also a huge mistake to get your science news and opinion from The Economist--a magazine with a great reputation that has nothing to do with its coverage of science

Re: Why most published scientific research is probably false [video]

#44

Post by snowwrestler »

001sky wrote: post #15

Two game-theoretic strategies need to be mitigated/bred out out of Academia: (1)'Security through obscurity' problem, where nobody can be bothered to verify your results as they are likely meaningless, lack broad applicability, or are not intellectally cost-effective for anyone to be bothered to understand them (etc). (2) The "lick the cookie" problem, where nobody will verify your results because there its considere…

Neither of these are actually problems.

If (1), your results are obscure or lack applicability, then they are not useful anyway and so it doesn't matter if they are ever confirmed. A surprisingly large amount of scientific research fits into this category.

While (2) few researchers set out solely to verify another's result, if the result is noteworthy, it will be incorporated as a component of future research and therefore tested indirectly. This other comment (not mine) explains how:

https://news.ycombinator.com/item?id=6662124

Re: Why most published scientific research is probably false [video]

#45

Post by a_bonobo »

capnrefsmmat wrote: post #40
a_bonobo wrote: post #9

Earlier quoted context omitted.

There's a follow-up paper to the one you linked that claims that much less published research findings are false: >We estimate that the overall rate of false discoveries among reported results is 14% (s.d. 1%), contrary to previous claims. We also found that there is no a significant increase in the estimated rate of reported false discovery results over time (0.5% more false positives (FP) per year, P=0.18) or with…

You should also read the responses to that article, such as Ioannidis's, which is scathing. They're all open-access, thankfully, and you can find them here: http://simplystatistics.org/2013/09/25/is-most-science-false... Jager and Leek put their code on GitHub, so the commenters were able to review their code and tinker with it to see what would happen.

Thanks for all these links! I skimmed them, and it's fun to see that two of them rip into Ionnadis' work, while Ionnadis rips into the Jager/Leek work... It's also a cool exercise in reproducible research and open peer review, both of which are far from common.

In my opinion, the entire exercise of data-mining the published literature is pretty much futile. We already know there are problems in the published literature and that scientists are pretty mediocre at statistics. Pin-pointing the exact value of how mediocre only leads to, as your link shows, a mountain of published works, hurt feelings (on Ionnadis side I guess) and doesn't solve anything.

Re: Why most published scientific research is probably false [video]

#46

Post by vixen99 »

gabriel34 wrote: post #32

IMO what is damaging to society is that there is no PR to science, so the press takes things that are not yet fully understood or verified by the academic community and publishes it as confirmed (studies say, confirm, etc.) Groundbreaking results give much more press than their rebuttal, for example, see neutrinos faster than light or that arsenic consuming bacteria both of which were later dismissed in academic circ…

I can't imagine what process you have in mind to put right what you see as a problem. Who is the arbiter of the press coverage? You are assuming unrealistic habits on the part of the passive observer of the results of research. Life isn't spent checking out media coverage or providing it with checks and balances unless there's a major issue involved which impinges on people's lives. The solution to your perceived difficulty is to check the facts for yourself from original sources and not expect some arbitrary secondary source to keep you up to date because that is something they are never going to do.

Re: Why most published scientific research is probably false [video]

#47

Post by leoc »

gabriel34 wrote: post #32

IMO what is damaging to society is that there is no PR to science, so the press takes things that are not yet fully understood or verified by the academic community and publishes it as confirmed (studies say, confirm, etc.) Groundbreaking results give much more press than their rebuttal, for example, see neutrinos faster than light or that arsenic consuming bacteria both of which were later dismissed in academic circ…

That sounds very much like the UK's Science Media Centre: http://www.sciencemediacentre.org/about-us/ . How good a thing the SMC has been is another question.

Re: Why most published scientific research is probably false [video]

#48

Post by bnegreve »

The claim is:

    "most published scientific research is probably false"
and the evidence for that claim is:

    The number of false negative "might easily be 4 in 10, or in some fields 8 in 10"
(quoted from the video)

This is rather weak.

And maybe more importantly: this assumes that researchers test random hypothesis uniformly drawn in the space of all possible hypothesis, which clearly isn't the case.

Anyway, this can't be really serious.

Re: Why most published scientific research is probably false [video]

#49

Post by pygy_ »

epistasis wrote: post #34
001sky wrote: post #30

Earlier quoted context omitted.

If she has not been doing something interesting, that line of research does not get cited, dies out, and does not get funding. Could you explain how this verifies results? Citation (alone) isn't verification. And its safe to say followers and funding correlate with one another, at least statistically.

You're reversing what I'm trying to say, I'm saying that verification or invalidation causes citations to occur, not that citation is verification. And that work that doesn't get validated dies out. Even work that gets "invalidated" in some sense can reveal other truths, or necessary changes in paradigms. This is why scientists check the citations of a paper when considering its contents. It's important to ask if oth…

Work often gets cited without being either replicated or invalidated.

Do you do any kind of research? You seem to have an unrealistically rosy perception of the scientific process as it happens concretly.

Re: Why most published scientific research is probably false [video]

#50

Post by eli_gottlieb »

Fomite wrote: post #20
epistasis wrote: post #16

The vast majority of scientific papers are not single experiments with one p-vaule, but rather a handful experiments to a dozen or more experiments, only some of which may be reduced to a p-value. And in most biological research, at least two lines of evidence are required before a reviewer will accept a claim (e.g. "OK, you may have found something, now verify it with a PCR."). So this entire setup is just kind of c…

What's infuriatingly ignored is that in that very same PLoS Medicine issue is a response to Ioannidis' work by Greenland, IIRC, that notes that by "False" he means the significance is wrong, but what's really of interest is the effect measure. On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

>On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

Because in science, not believing things is the default state. If you say, "most published findings are false", you're really saying, "most of the time we have to accept the null hypothesis", which is what we all not-so-secretly believe regarding everything, all the time, in any case.

Post reply on HN