Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Most Published Research Findings Are False

ncbi.nlm.nih.gov

Post reply on HN
Print view View original

41–45 of 45 posts

Re: Why Most Published Research Findings Are False

#41

Post by tokenadult »

St-Clock wrote: post #38
jonhendry wrote: post #6

Well, that's why the whole "replication" thing is important. One published result is interesting, but rarely definitive, and possibly wrong. (Or at least unusual for possibly difficult-to-determine reasons.) This is another good reason to ignore the media hype for every new paper that comes out. (Besides the fact that journalists perform lossy compression on data.) But it seems like it's how science is supposed to wo…

On the whole, I agree with you, but replication is often expensive and in some fields (e.g., software engineering research), it's not enough to warrant a publication in a high profile journal or conference (especially if you confirm the original findings). Heck, you may not even get funding to do that. Regarding "if you seem to have done everything correctly", I don't think that any honest scientist can claim that hi…

Regarding "if you seem to have done everything correctly", I don't think that any honest scientist can claim that his/her study had no limitation or flaw.

Hear. Hear. This, of course, was the thrust of Richard Feynman's famous Caltech commencement speech on "Cargo Cult Science."

http://www.lhup.edu/~DSIMANEK/cargocul.htm (adapted HTML text)

http://calteches.library.caltech.edu/51/2/CargoCult.pdf (PDF version as published by Caltech)

As Feynman said, "The first principle is that you must not fool yourself--and you are the easiest person to fool. So you have to be very careful about that."

Re: Why Most Published Research Findings Are False

#42

Post by michael_nielsen »

llimllib wrote: post #7

The author's research is discussed in an excellent Atlantic article that many may find more accessible: http://www.theatlantic.com/magazine/archive/2010/11/lies-dam...

There's also a nice short summary in this blog post:

http://www.marginalrevolution.com/marginalrevolution/2005/09...

Re: Why Most Published Research Findings Are False

#44

Post by alan-crowe »

elbenshira wrote: post #4

First of all, the author of this piece works in Department of Hygiene and Epidemiology. Research is done differently across different disciplines, so it's dangerous to try to expand this to other disciplines. For example, some fields find alpha But research is very weird indeed. The more conference/journal articles you read, the less you trust them. I mean, say a field accepts results alpha Feel free to correct me if…

The traditional notion of a 5% confidence limit comes from devising rules of thumb for agricultural research stations in the 1930's. The basic frame work is that each experiment takes many months, a large plot of land, and plenty of money. You test crop varieties that you are already confident will give a better yield in order to check that they really do so.

Suppose your initial guessing is 50:50 and over some years you run 200 tests. 100 times the crop really does yield better and most of those show up fine. 100 times the crop doesn't actually yield better and 5% of those result in false positives. You end up with around 100 true positives and 5 false positives. A positive result really means something.

Fast forward 80 years and research has changed. You have high throughput screening machines and can test 100,000 different molecules in your hunt for a new antibiotic. Suppose you have got lucky and there really is a new antibiotic in your combinatorial explosion of side chains. A p-value of 5% gives you 5000 false positives. With any luck you don't get a false negative and your new antibiotic also makes it through the initial screen. Now you have 5001 +/- 70 positives. The probability that a positive result is true is only 0.0002 or 0.02%. A positive results still means something important. You are searching for a needle in a haystack and you have discarded 95% of the hay, but there is still plenty of hay left and the 99.98% of the results are wrong.

Re: Why Most Published Research Findings Are False

#45

Post by jonhendry »

St-Clock wrote: post #38
jonhendry wrote: post #6

Well, that's why the whole "replication" thing is important. One published result is interesting, but rarely definitive, and possibly wrong. (Or at least unusual for possibly difficult-to-determine reasons.) This is another good reason to ignore the media hype for every new paper that comes out. (Besides the fact that journalists perform lossy compression on data.) But it seems like it's how science is supposed to wo…

On the whole, I agree with you, but replication is often expensive and in some fields (e.g., software engineering research), it's not enough to warrant a publication in a high profile journal or conference (especially if you confirm the original findings). Heck, you may not even get funding to do that. Regarding "if you seem to have done everything correctly", I don't think that any honest scientist can claim that hi…

"Regarding "if you seem to have done everything correctly", I don't think that any honest scientist can claim that his/her study had no limitation or flaw."

Right, that's what I was getting at. The scientist might believe they've done everything right, after checking and re-checking their work, but be missing some flaw or limitation in their work, their model, whatever.

I recall hearing once of an experiment that couldn't be reproduced, and it turned out to be due to some chemical property of the entirely normal laboratory glassware that one lab had used. Switching to another manufacturer removed the problem. (I'm probably messing up the details, like the consequences of the chemical properties of the glass. But the gist is correct. Different manufacturer of glassware cleared up a problem that was unexpected.)

Post reply on HN