Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why most published scientific research is probably false [video]

economist.com

Post reply on HN
Print view View original

31–40 of 57 posts

Re: Why most published scientific research is probably false [video]

#31

Post by dhaivatpandya »

pygy_ wrote: post #17
chasing wrote: post #10

Look. Clearly science makes progress that's "true" in the sense that it becomes useful and can be used as functional models of the way things work. This video's using a very reductionist kind of statistics to point out that, yes, an individual piece of research making a claim might have a good chance of being wrong. Which is why science doesn't say, "Oh, well Larry just proved that Saturn orbits Uranus so let's just…

> Science checks itself. Not that much, actually. There's no point in reproducing the work of others to advance your own scientific career. Unless your research directly directly depends on them, it is counterproductive.

Agreed. Try getting funding for verifying a paper's results. If you can't publish results, it'll be a hard time convincing anyone that your work is worthwhile.

Re: Why most published scientific research is probably false [video]

#32

Post by gabriel34 »

IMO what is damaging to society is that there is no PR to science, so the press takes things that are not yet fully understood or verified by the academic community and publishes it as confirmed (studies say, confirm, etc.)

Groundbreaking results give much more press than their rebuttal, for example, see neutrinos faster than light or that arsenic consuming bacteria both of which were later dismissed in academic circles, but did not enjoyed the same treatment from the media.

Re: Why most published scientific research is probably false [video]

#33

Post by tga_d »

gabriel34 wrote: post #26
pygy_ wrote: post #17

Earlier quoted context omitted.

> Science checks itself. Not that much, actually. There's no point in reproducing the work of others to advance your own scientific career. Unless your research directly directly depends on them, it is counterproductive.

Actually there are loads of scientific papers only reviewing other papers (usually a set of them), trying to infer conclusions and criticizing them on methodology and rigidity. Scientific careers advance both in quality and in quantity of papers published. Not everyone gets featured on Science or Nature.

This is actually what I'm doing for my undergrad senior thesis right now. It's in theory-heavy CS, but there are some results that are based on some moderately difficult math, and experimental evidence of those results, which I'm independently verifying. So, it seems to apply. It's not the most exciting thing to be doing in the field, but it's more interesting than most of my other classes, and can be used for later work.

Re: Why most published scientific research is probably false [video]

#34

Post by epistasis »

001sky wrote: post #30
epistasis wrote: post #21

Earlier quoted context omitted.

In the important areas, it does check itself. If a scientist has been doing some interesting work, she will soon find herself with colleagues working on the same area. If she has not been doing something interesting, that line of research does not get cited, dies out, and does not get funding. In active learning, there's a tradeoff between exploration and exploitation. Science does this too, but heuristically.

If she has not been doing something interesting, that line of research does not get cited, dies out, and does not get funding. Could you explain how this verifies results? Citation (alone) isn't verification. And its safe to say followers and funding correlate with one another, at least statistically.

You're reversing what I'm trying to say, I'm saying that verification or invalidation causes citations to occur, not that citation is verification. And that work that doesn't get validated dies out. Even work that gets "invalidated" in some sense can reveal other truths, or necessary changes in paradigms.

This is why scientists check the citations of a paper when considering its contents. It's important to ask if other people followed up on this, and what have they found if they did follow up? New papers, without any citations, must be held in a state of meta-information, until there's follow up papers. Old papers with few citations, and no validation citations, must also be considered as in a state of meta-information. Sometimes, really important things like Mendel's genetics get lost for decades in this state, until they are rediscovered, but it's fairly rare.

Re: Why most published scientific research is probably false [video]

#35

Post by bloaf »

001sky wrote: post #30
epistasis wrote: post #21

Earlier quoted context omitted.

In the important areas, it does check itself. If a scientist has been doing some interesting work, she will soon find herself with colleagues working on the same area. If she has not been doing something interesting, that line of research does not get cited, dies out, and does not get funding. In active learning, there's a tradeoff between exploration and exploitation. Science does this too, but heuristically.

If she has not been doing something interesting, that line of research does not get cited, dies out, and does not get funding. Could you explain how this verifies results? Citation (alone) isn't verification. And its safe to say followers and funding correlate with one another, at least statistically.

Say someone publishes a paper that has the conclusion: If you preform procedure X, you get carbon nanotubes with property Y.

Lets then go on to say that someone publishes a paper which states: carbon nanotubes with property Y will theoretically allow us to do Z.

If Z is something interesting and worthwhile, it won't be long before people read both papers and decide to try using procedure X to make Y nanotubes and then measure their ability to do Z. If no one can actually get to Z, then one or both of those papers was wrong, and the field will recognize that. The papers may not be formally retracted, but everyone will move on and stop referencing the incorrect papers.

Re: Why most published scientific research is probably false [video]

#36

Post by xherberta »

001sky wrote: post #15

Two game-theoretic strategies need to be mitigated/bred out out of Academia: (1)'Security through obscurity' problem, where nobody can be bothered to verify your results as they are likely meaningless, lack broad applicability, or are not intellectally cost-effective for anyone to be bothered to understand them (etc). (2) The "lick the cookie" problem, where nobody will verify your results because there its considere…

Interesting, and nicely stated! I wonder what it would look like if funding and publishing could be made to incentivize really valuable pursuits, rather than rewarding trendiness, novelty, and shock value.

Re: Why most published scientific research is probably false [video]

#37

Post by capnrefsmmat »

epistasis wrote: post #21
pygy_ wrote: post #17

Earlier quoted context omitted.

> Science checks itself. Not that much, actually. There's no point in reproducing the work of others to advance your own scientific career. Unless your research directly directly depends on them, it is counterproductive.

In the important areas, it does check itself. If a scientist has been doing some interesting work, she will soon find herself with colleagues working on the same area. If she has not been doing something interesting, that line of research does not get cited, dies out, and does not get funding. In active learning, there's a tradeoff between exploration and exploitation. Science does this too, but heuristically.

Not necessarily true. Ioannidis did a review of 49 studies cited more than 1,000 times in the medical literature -- prime candidates for being replicated or tested by future results. Of those,

- 16% were found by later studies to be wrong

- 16% were found to have exaggerated the size of effects they claimed to detect

- 44% were replicated, and

- 24% were unchallenged and unreplicated by later literature.

So a full quarter of incredibly prominent research was never tested.

Ioannidis, J. P. A. (2005). Contradicted and initially stronger effects in highly cited clinical research. JAMA, 294(2), 218–228. doi:10.1001/jama.294.2.218

Re: Why most published scientific research is probably false [video]

#38

Post by capnrefsmmat »

Fomite wrote: post #20
epistasis wrote: post #16

The vast majority of scientific papers are not single experiments with one p-vaule, but rather a handful experiments to a dozen or more experiments, only some of which may be reduced to a p-value. And in most biological research, at least two lines of evidence are required before a reviewer will accept a claim (e.g. "OK, you may have found something, now verify it with a PCR."). So this entire setup is just kind of c…

What's infuriatingly ignored is that in that very same PLoS Medicine issue is a response to Ioannidis' work by Greenland, IIRC, that notes that by "False" he means the significance is wrong, but what's really of interest is the effect measure. On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

It's true that effect sizes are often more important, but it's also true that they're also often incorrect. See e.g.

Ioannidis, J. P. A. (2008). Why Most Discovered True Associations Are Inflated. Epidemiology, 19(5), 640–648. doi:10.1097/EDE.0b013e31818131e7

Most studies are underpowered and are incapable of detecting the true effect. Only if they get lucky and observe an abnormally large effect will they obtain a statistically significant result, so the published results tend to be significant overestiates.

For another good example, see

Gelman, A., & Weakliem, D. (2009). Of beauty, sex, and power: statistical challenges in estimating small effects. American Scientist, 97, 310–316.

http://www.stat.columbia.edu/~gelman/research/unpublished/po...

Re: Why most published scientific research is probably false [video]

#39

Post by capnrefsmmat »

lvs wrote: post #19

This is extremely misleading and feeds an anti-intellectual notion that scientists are just lying to everybody. First, it perpetuates a common claim of those who don't practice any sort of science: that the output of scientific studies is an enumeration of true/false claims determined with statistical inference logic. (Science media/blogs really aren't helping on this one.) Second, the math is just wrong: the space o…

I don't think the technicalities of the size of the hypothesis space or its uncountability matter. We do studies over large numbers of hypotheses, of which we know a priori only a few are likely to be true. Genome-wide association studies, for example, will study thousands of genes knowing that only a few will have true correlations.

In any case it is an entirely reasonable assumption that of all tested scientific hypotheses, a small fraction are true. We often do not expect them to be true. (A clinical trial may test thirty or more hypotheses, such as "this medication causes diarrhea," and may expect most of them to be false. But they have to check just in case.)

Re: Why most published scientific research is probably false [video]

#40

Post by capnrefsmmat »

a_bonobo wrote: post #9
bigchewy wrote: post #6

Earlier quoted context omitted.

yes, probably false. This is not news, e.g. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/ came out in 2005. of course, research "proving" that research tends to be false can get us into a little circular game...

There's a follow-up paper to the one you linked that claims that much less published research findings are false: >We estimate that the overall rate of false discoveries among reported results is 14% (s.d. 1%), contrary to previous claims. We also found that there is no a significant increase in the estimated rate of reported false discovery results over time (0.5% more false positives (FP) per year, P=0.18) or with…

You should also read the responses to that article, such as Ioannidis's, which is scathing. They're all open-access, thankfully, and you can find them here:

http://simplystatistics.org/2013/09/25/is-most-science-false...

Jager and Leek put their code on GitHub, so the commenters were able to review their code and tinker with it to see what would happen.

Post reply on HN