Odds Are, It's Wrong
sciencenews.org
Odds Are, It's Wrong
1–8 of 8 posts
Re: Odds Are, It's Wrong
#2I'm somewhat amused to see people complain that "commonly accepted" alphas of .05 are yielding high error rates. Surprise: setting your confidence interval at one-sigma yields a ton of false positives. There's a reason particle physics uses three-sigma as a threshold for further investigation, and five-sigma for discovery.
The difficulty is that in biology, psychology, and other really complicated systems, attaining even two or three sigma can be difficult and expensive. That suggest that we should continue to use low-confidence results... but accept their tentative nature instead of bestowing them with the aura of truth.
And before you start treating a few hundred thousand people with a drug, get a bigger sample size. :)
Re: Odds Are, It's Wrong
#3I'm somewhat amused to see people complain that "commonly accepted" alphas of .05 are yielding high error rates. Surprise: setting your confidence interval at one-sigma yields a ton of false positives. There's a reason particle physics uses three-sigma as a threshold for further investigation, and five-sigma for discovery. The difficulty is that in biology, psychology, and other really complicated systems, attaining…
That isn't the only problem addressed.
Misinterpretation and/or abuse of statistics is prevalent in scientific literature more often then you would think. Repeatability, consensus and peer review are good ways of minimizing this problem but it is still true that any random paper you open on say arXiv or even a standard journal is probably going to contain some form of abuse intentional or not.
Is this a serious problem? Mostly in those fields plagued by low confidence results and heavy approximations being made but it exists in all levels.
The article is rather alarmist though for my taste.
Re: Odds Are, It's Wrong
#4One thing to note is that there's a ton of selection bias: noteworthy findings get published, non noteworthy ones do not. Therefore scientists will submit papers containing noteworthy findings.
If you do 1,000 experiments in your lab, and (a) absolutely nothing interesting happens; (b) you get a naturally noisy dataset on each, and (c) you write up only those phenomena which would occur randomly only 5% of the time
... you will submit 50 articles, each of them claiming something noteworthy, and all of them false.
Perhaps there are a few actually noteworthy things that were discovered in your lab. ...in which case in addition to the 50 BS papers, you'll also submit 1 or 2 that describe some real effect.
Re: Odds Are, It's Wrong
#5A great article on how the Naymen-Pearson Hypothesis Testing is not particularly useful as a method of inference (e.g. the way most people tend to think of statistics with p-values and 5% significance doesn't say much). Some of what he is saying is a little shoddy, since Fisher actually came up with a much more robust, but less used, method of inference called Maximum likelihood estimation (MLE), which currently stands with The Bayesian Theory as the competing (yet partially complimentary) theories of statistical inference. Strict, but arbitrary, distinction between the null H0 and alternative H1 hypotheses, which is what p-values suggest, is as careless as the article points out.
Re: Odds Are, It's Wrong
#6I'm somewhat amused to see people complain that "commonly accepted" alphas of .05 are yielding high error rates. Surprise: setting your confidence interval at one-sigma yields a ton of false positives. There's a reason particle physics uses three-sigma as a threshold for further investigation, and five-sigma for discovery. The difficulty is that in biology, psychology, and other really complicated systems, attaining…
That isn't the only problem addressed. Misinterpretation and/or abuse of statistics is prevalent in scientific literature more often then you would think. Repeatability, consensus and peer review are good ways of minimizing this problem but it is still true that any random paper you open on say arXiv or even a standard journal is probably going to contain some form of abuse intentional or not. Is this a serious probl…
Oh, yeah, the 5% thing is only a small part of the article. I've just read four or five "OMG 80% of science is wrong!" articles this month, and think it's kinda tired.
Re: Odds Are, It's Wrong
#7This is an interesting critique, but equating science with "a crapshoot" is journalistically irresponsible. IANAS, but I read a few science blogs, and the impression I get is that most scientific papers hedge and qualify extensively, laboriously mentioning statistical significance, noting possible alternative explanations for the observed data and imagining future experiments that ought to be run to provide extra confidence in the findings.
He gives the example of all the studies linking genetic variants to acute coronary syndrome, only one of which was confirmed in a later (presumably larger?) study. Well, what does it mean for a study to link a gene to a condition? Maybe they concluded, "gene FOO19 causes heart attacks, bitches!" Or maybe it was more like, "4 out of 5 coronary sufferers we studied had this FOO19 gene. Someone should look into it." Those are two very different statements; one is way out-of-line, and the other is responsible reportage.
Re: Odds Are, It's Wrong
#8I see no one commenting on this interesting article has yet mentioned an article on experiment design by a scientist-hacker, Peter Norvig,
http://norvig.com/experiment-design.html
that I mention frequently in comments here on HN because
1) it is accessible to general readers,
2) it is written by someone with excellent formal training and work experience in the subject,
3) it links to excellent primary source literature for further reading,
and
4) it gives cogent, thought-provoking examples relating to errors of reasoning that are common even among trained scientists.
Because I like science, and I like science to progress based on firmly verified factual conclusions, I like my friends here on HN to know about the pitfalls to look out for when anyone does experimental research. Thanks for the submission of the latest journalistic article on related issues here, which also gives us a lot to think about and which has elicited informative comments.