So if this research is not false (unlikely according to the author) then mankind would be moving backwards, unless non scientific reasoning compensates for the failure of science. Medical treatment would get constantly worse, people would be misdiagnosed and mistreated more than ever, death rates after cancer and cardiac events would rise.
Why Most Published Research Findings Are False
31–40 of 45 posts
Re: Why Most Published Research Findings Are False
#32So if this research is not false (unlikely according to the author) then mankind would be moving backwards, unless non scientific reasoning compensates for the failure of science. Medical treatment would get constantly worse, people would be misdiagnosed and mistreated more than ever, death rates after cancer and cardiac events would rise.
That's a lot of assumptions for 2 sentences.
Re: Why Most Published Research Findings Are False
#33Earlier quoted context omitted.
That's not the case, even the most widely cited research is dubious. From the Atlantic article: "He zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these jo…
Were some of those citations made to prove those findings wrong? Dumb question, but would be a shame to not be thourough when attacking bad literature.
Yeah, that's a commonly mentioned problem with citation-counting. Much like linkbaiting on the internet, a poor-quality paper taking an inflammatory position can get a lot of citations from people debunking it. Another problem is throwaway citations: some paper gets cited as a generic example, rather than because it provides anything valuable that the paper citing it actually draws on.
Unfortunately, it's much harder to come up with better measures. Given a smallish corpus of a few hundred papers, humans could read through them and annotate each citation with things like, "cited to debunk", "cited to distinguish related work", "cited for general background", "cited in passing", "cited for result", etc. But computers are not yet very good at doing that automatically, so the large-scale citation analysis just does dumb citation-counting.
Re: Why Most Published Research Findings Are False
#34Earlier quoted context omitted.
You will have to use binomial distribution ( http://en.wikipedia.org/wiki/Binomial_distribution ) for finding the probability that 5% of them are wrong (Assuming researches are independent). I guess this probability will come to be very small. But the point of this article is not alpha(confidence level) ,It is the bias of the researcher.
p-values are calculated in many different ways, not always using the binomial distribution (of the p-values I've claimed, very few have used the binomial distribution)
i was referring to the chances of 5% of the papers giving wrong results. using pvalue as the chance of failure in each research paper , binomial can be used to find the probability that 5% of the papers are in correct
Re: Why Most Published Research Findings Are False
#35Its not surprising that misrepresentation has become so common in what is now a largely political industry.
Scientists gotta eat like everyone else, they will do what they need to do to get funding. It's why so much "climate science" is dodgy, the sky is falling! type research gets more headlines. Witness the recent report on the Himalayas melting for a perfect example of this phenomenon at work.
Re: Why Most Published Research Findings Are False
#36So if this research is not false (unlikely according to the author) then mankind would be moving backwards, unless non scientific reasoning compensates for the failure of science. Medical treatment would get constantly worse, people would be misdiagnosed and mistreated more than ever, death rates after cancer and cardiac events would rise.
That's a lot of assumptions for 2 sentences.
Those are implications of what that study claims to have found, not to be taken entirely serious ;-)
Re: Why Most Published Research Findings Are False
#37So if this research is not false (unlikely according to the author) then mankind would be moving backwards, unless non scientific reasoning compensates for the failure of science. Medical treatment would get constantly worse, people would be misdiagnosed and mistreated more than ever, death rates after cancer and cardiac events would rise.
That's a lot of assumptions for 2 sentences.
[deleted]
Re: Why Most Published Research Findings Are False
#38Well, that's why the whole "replication" thing is important. One published result is interesting, but rarely definitive, and possibly wrong. (Or at least unusual for possibly difficult-to-determine reasons.) This is another good reason to ignore the media hype for every new paper that comes out. (Besides the fact that journalists perform lossy compression on data.) But it seems like it's how science is supposed to wo…
On the whole, I agree with you, but replication is often expensive and in some fields (e.g., software engineering research), it's not enough to warrant a publication in a high profile journal or conference (especially if you confirm the original findings). Heck, you may not even get funding to do that.
Regarding "if you seem to have done everything correctly", I don't think that any honest scientist can claim that his/her study had no limitation or flaw. I regularly review papers for big conferences and there is no such thing as a perfect paper/research project/study. It's more like a threshold: despite the issues, were the findings novel, relevant, and found through a rigorous process? Would the community learn anything valuable by reading this?
Articles like the one cited by OP are useful if they make scientists and normal "folks" realize the limitations of alpha values, they are useful if they make scientists reconsider some of their methods (and way of presenting findings), but they can be harmful if the readers conclude that most scientific findings are "false" and thus, that science is bogus because it cannot find the "truth". Science is rarely, if ever, about true and false, religion is.
P.S. I realize this answer was more about the article, and less about your reply, it's just that your reply prompted me to write something :-) Again, I agree with you!
Re: Why Most Published Research Findings Are False
#39This paper has gotten way too much press for an oversimplified model of science. Here's the thing: if results hold up to scrutiny, the authors are eager to share code and plasmids/samples. If not, they are a lot more squirrelly. Outside replication is what keeps the machine moving forward, is fairly readily proxied by citation rates, and yet is not captured by Ioannidis' simple model.
[deleted]
Re: Why Most Published Research Findings Are False
#40In statistics, you're supposed to come up with a statistical model first before running regressions on the data. But quite a few papers I've read (especially in finance) seem to go the other way around, i.e. They run regressions on a data set, adding and subtracting independent variables until the t values and standard errors start looking good. Then they construct the linear model, assume the Gauss-Markov assumption…
there's nothing wrong with looking at some of the data first per se, provided you do not use the same data to draw conclusions. i.e., have a training and a test data set.