Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why most published scientific research is probably false [video]

economist.com

Post reply on HN
Print view View original

51–57 of 57 posts

Re: Why most published scientific research is probably false [video]

#51

Post by eli_gottlieb »

001sky wrote: post #15

Two game-theoretic strategies need to be mitigated/bred out out of Academia: (1)'Security through obscurity' problem, where nobody can be bothered to verify your results as they are likely meaningless, lack broad applicability, or are not intellectally cost-effective for anyone to be bothered to understand them (etc). (2) The "lick the cookie" problem, where nobody will verify your results because there its considere…

>(1)'Security through obscurity' problem, where nobody can be bothered to verify your results as they are likely meaningless, lack broad applicability, or are not intellectally cost-effective for anyone to be bothered to understand them (etc).

This is simply a problem of explaining findings to other researchers. Every single researcher with a genuine finding on his hands will, at some point, seem like his work is obscure and pointless compared to "all those other guys" who are building on established research for immediate application.

And then there's the issue that your failure to understand someone's research doesn't mean it isn't research. Sometimes what a field needs is just a few more smart people willing to work at understanding what the hell's going on.

Re: Why most published scientific research is probably false [video]

#53

Post by tokenadult »

Fomite wrote: post #20
epistasis wrote: post #16

The vast majority of scientific papers are not single experiments with one p-vaule, but rather a handful experiments to a dozen or more experiments, only some of which may be reduced to a p-value. And in most biological research, at least two lines of evidence are required before a reviewer will accept a claim (e.g. "OK, you may have found something, now verify it with a PCR."). So this entire setup is just kind of c…

What's infuriatingly ignored is that in that very same PLoS Medicine issue is a response to Ioannidis' work by Greenland, IIRC, that notes that by "False" he means the significance is wrong, but what's really of interest is the effect measure. On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

This is a fair question. I think the reasons the Ioannidis paper was persuasive are that

1) Ioannidis replicated earlier results about the lack of replication of most research reports,

and

2) Ioannidis "showed the work" for how possible, and indeed likely, it is for an effect size that permits a false-positive finding to be published, under reasonable assumptions about the prevalence of false-positive findings and publishing practices. Most scientists were vaguely aware of lack of replication years before anyone heard of Ioannidis, but not many scientists were fully aware of how readily a false-positive finding can be published.

Re: Why most published scientific research is probably false [video]

#54

Post by officialjunk »

chattoraj wrote: post #14
officialjunk wrote: post #5

so... non-scientific articles about scientific research are true? probably not.

See the original paper: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/

i see. but then again, wouldn't their findings imply their own published results are likely false?

Re: Why most published scientific research is probably false [video]

#55

Post by epistasis »

pygy_ wrote: post #49
epistasis wrote: post #34

Earlier quoted context omitted.

You're reversing what I'm trying to say, I'm saying that verification or invalidation causes citations to occur, not that citation is verification. And that work that doesn't get validated dies out. Even work that gets "invalidated" in some sense can reveal other truths, or necessary changes in paradigms. This is why scientists check the citations of a paper when considering its contents. It's important to ask if oth…

Work often gets cited without being either replicated or invalidated. Do you do any kind of research? You seem to have an unrealistically rosy perception of the scientific process as it happens concretly.

>Work often gets cited without being either replicated or invalidated.

I'm not sure why you're bringing up this fact, which is not at all inconsistent with my comment. What's your implication?

>Do you do any kind of research? You seem to have an unrealistically rosy perception of the scientific process as it happens concretely.

In the past decade, I've spent probably 70% of my time on scientific research. At this moment I'm procrastinating from writing a letter of support on a grant. What, in particular, do you think I'm unrealistically rosy about?

Re: Why most published scientific research is probably false [video]

#56

Post by Fomite »

capnrefsmmat wrote: post #38
Fomite wrote: post #20

Earlier quoted context omitted.

What's infuriatingly ignored is that in that very same PLoS Medicine issue is a response to Ioannidis' work by Greenland, IIRC, that notes that by "False" he means the significance is wrong, but what's really of interest is the effect measure. On a meta level, I've always wondered why we take a paper about most findings being false as clearly correct.

It's true that effect sizes are often more important, but it's also true that they're also often incorrect. See e.g. Ioannidis, J. P. A. (2008). Why Most Discovered True Associations Are Inflated. Epidemiology, 19(5), 640–648. doi:10.1097/EDE.0b013e31818131e7 Most studies are underpowered and are incapable of detecting the true effect. Only if they get lucky and observe an abnormally large effect will they obtain a s…

I think part of the point there is not to pass effect estimates through a significance test filter first. Most studies are underpowered to detect a true effect at alpha = 0.05. That doesn't actually suggest that most studies are wrong as much as if a study is underpowered and doesn't find a significant finding, we assert its dull and uninteresting.

Ironically, the Ioannidis paper is in Epidemiology, which is a journal that is fairly anti-significance testing, but where I still get reviewers suggesting that an effect measure with a confidence interval that brushes against the null must mean nothing at all.

Re: Why most published scientific research is probably false [video]

#57

Post by pygy_ »

epistasis wrote: post #55
pygy_ wrote: post #49

Earlier quoted context omitted.

Work often gets cited without being either replicated or invalidated. Do you do any kind of research? You seem to have an unrealistically rosy perception of the scientific process as it happens concretly.

>Work often gets cited without being either replicated or invalidated. I'm not sure why you're bringing up this fact, which is not at all inconsistent with my comment. What's your implication? >Do you do any kind of research? You seem to have an unrealistically rosy perception of the scientific process as it happens concretely. In the past decade, I've spent probably 70% of my time on scientific research. At this mom…

> I'm not sure why you're bringing up this fact, which is not at all inconsistent with my comment. What's your implication?

In that case I may have misunderstood your point. What I mean is that, for a paper with 100+ citations (which, in some fields, is not rare), most of them are not verified by the authors.

> In the past decade, I've spent probably 70% of my time on scientific research. At this moment I'm procrastinating from writing a letter of support on a grant. What, in particular, do you think I'm unrealistically rosy about?

The self-correcting nature of the process is very slow in most cases. Bad results end up being forgotten for minor findings, but for things of mild interest, it may linger far longer.

Post reply on HN