Science’s Biggest Fail

I put my hand in a fire, I get a burn right away. Pretty easy to know what happened. Sure there were many variables. My shoes, my pants, the weather.. but pretty clear.

Drinking a drink a day may do something good. May do something bad. But both those events happen in 20 years. Holy crap that is hard to tell! (but yes, partially because it mixes with many variables).

There is also a part of science of "we all WANT what makes us happy to be deemed healthy!". Decision affirmation rocks! So if there are 4 studies that say a drink a day is bad for you, and 2 that say a drink a day is good - which makes the newspaper? Coffee, chocolate, alcohol - all things people love, and all affirmed by at least SOME studies out there. Awesome!

a) Is the data made available? b) Is it a Bayesian analysis? c) Has a power study been offered?

As a statistician, I have a keen awareness of the ways that p-values can depart from truth. You can see Optimizely's effort to cope (https://www.optimizely.com/statistics). You can read about it in The Cult of Statistical Significance (http://www.amazon.com/The-Cult-Statistical-Significance-Econ...). This Economist video captures it solidly (http://www.economist.com/blogs/graphicdetail/2013/10/daily-c...).

The key component missing is a bias towards positive results. Most scientists only have two statistics classes. In these classes they learn a number of statistical tests, but much less how things can go wrong. Classic, "just enough to be dangerous."

In order to cope, I have a personal set of criteria to make a quick first sort of papers. It's a personal heuristic for quality. I assume some degree of belief (Bayes, FTW!) that those that offer the full data set along side conclusions feel confident in their own analysis. Also, if they're using Bayesian methods, that they've had more than two stats classes. Finally, if they do choose Frequentist methods, a power study tells me that they understand the important finite nature of data in the context of asymptotic models / assumptions.

I'd suspect that other statisticians feel this way, because I've heard that privately --- what do you think of my criteria?

The problem is media. News programming is constantly looking for something to breathlessly report, and is delighted to find one or more so-called experts who will loudly extol / lambaste the latest findings.

If you look at the leading causes of death, understand your own history and risks, and follow the advice of credible doctors, you'll be doing great. Most of us don't do that, and then we scramble for fix-alls.

It's these silver-bullet, "all-x-is-bad, all-y-is-good, z-causes-cancer, w-cures-cancer," reports that are jerking your chain.

For every big "scientific" view that has an agenda behind it that either vastly overreaches the available research or lags decades behind the available research, you can generally point a finger at one or more corporations, government agencies, or not-for-profits.

GMO fearmongering? Largely environmentalist NFPs.

Breast cancer overdiagnosis? Largely cancer fundraising NFPs.

Nutritional science? Largely the government.

Let's tell pregnant women to live in a thick cocoon of craziness? Medical insurers.

And so forth.