Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistics Done Wrong – The woefully complete guide

refsmmat.com

Post reply on HN
Print view View original

11–20 of 71 posts

Re: Statistics Done Wrong – The woefully complete guide

#11

Post by pessimizer »

onion2k wrote: post #6
ultrasaurus wrote: post #5

One thing that constantly saddens me about statistics is that a large amount of energy is expended using is almost correctly to "prove" something that was already the gut feel. Even unbiased practitioners can be lead astray [1] but standards on how not to intentionally lie with statistics are very useful. [1] http://euri.ca/2012/youre-probably-polluting-your-statistics...

There's no way to tell whether or not that "gut feel" is accurate without proof. Often it's right, but occasionally it's very, very wrong (cancer risk and Bayes theory provides a good illustration: http://betterexplained.com/articles/an-intuitive-and-short-e... ). Consequently it's still worthwhile proving things even when they're seemly obvious.

I think his point was that people seem to "prove" common sense statistically all of the time - but when doing so make a lot of thoughtless assumptions about representativeness, significance, definitions, etc. stemming from the unspoken assumption of a particular outcome being inevitable.

Or maybe I'm projecting?

Re: Statistics Done Wrong – The woefully complete guide

#12

Post by neuralk »

dragonwriter wrote: post #9
neuralk wrote: post #4

Earlier quoted context omitted.

With respect to that Twain/Disraeli quote, my friend who is a professor of statistics tells me that he cannot go to a party and say what he does for a living without someone repeating it smirkingly.

Isn't that why the name "Data Science" was invented?

Data science sure sounds sexier...ish.

The irony of people who use the "damned lies and statistics" quote snidely is that the "statistics" part is not referring to the field Statistics but the plural version of the noun statistics, which of course are easily abused. The field of Statistics is all about NOT abusing statistics.

Re: Statistics Done Wrong – The woefully complete guide

#13

Post by glutamate »

One way to do statistics less wrong is to move from statistical testing to statistical modelling. This is what we are trying to support with BayesHive at https://bayeshive.com

Other ways of doing this include JAGS (http://mcmc-jags.sourceforge.net/) and Stan (http://mc-stan.org/)

The advantage of statistical modelling is that it makes your assumptions very explicit, and there is more of an emphasis on effect size estimation and less on reaching arbitrary significance thresholds.

Re: Statistics Done Wrong – The woefully complete guide

#14

Post by Homunculiheaded »

ultrasaurus wrote: post #5

One thing that constantly saddens me about statistics is that a large amount of energy is expended using is almost correctly to "prove" something that was already the gut feel. Even unbiased practitioners can be lead astray [1] but standards on how not to intentionally lie with statistics are very useful. [1] http://euri.ca/2012/youre-probably-polluting-your-statistics...

One of the functions of the prior in Bayesian analysis is to incorporate this "gut feel" into your calculations. Given that you have a strong prior belief and weak data (ie not much data) your belief will strongly influence the posterior. As you collect more data your belief will be increasingly overridden by the reality.

Re: Statistics Done Wrong – The woefully complete guide

#15

Post by jimmar »

One of the many challenges in science is that there is no publication outlet for experiments that just didn't pan out. If you do an experiment and don't find statistical significance, there aren't many journals that want to publish your work. That alone helps contribute to a bias toward publishing results that might have been found by chance. If 20 independent researchers test the same hypothesis, and there is no real effect, 1 might find statistical significance. That 1 researcher will get published. The 19 just move on.

Re: Statistics Done Wrong – The woefully complete guide

#16

Post by capnrefsmmat »

Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here.

I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a broad spectrum of people find it useful. I'd appreciate any suggestions from the HN crowd.

(A few folks have already emailed me with tips and suggestions. Thanks!)

(Also, I'm sure glad I added that email signup a couple weeks ago)

Re: Statistics Done Wrong – The woefully complete guide

#17

Post by Tycho »

If I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating the application of statistical methods being used; a replication division for checking that experiments do actually replicate; and a corruption division for investigating suspicious influences on the research. It would be tempting to then generate some sort of credibility rating for each institution based on the papers they're publishing, but that would probably invite too much trouble, so best just to publish the results and leave it at that.

Arguably this would be a greater benefit to humanity than all the millions poured charitably into cancer research etc.

Re: Statistics Done Wrong – The woefully complete guide

#19

Post by davmre »

Tycho wrote: post #17

If I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating the application of statistical methods being used; a replication division for checking that experiments do actually replicate; and a corruption division for investigating suspicious influences on the research. It would be tempting to then generate some sort of…

Something like that idea has actually already been the inspiration for at least one startup: MetaMed (http://en.wikipedia.org/wiki/MetaMed, http://nymag.com/health/bestdoctors/2013/metamed-personalize...) does meta-level analysis of the medical literature to determine which treatments seem effective for rare conditions, taking into account the sample size, statistical methodology, funding sources, etc. of each study.

Of course, medicine might be unique as a domain in which individuals are willing to pay vast sums of money to obtain slightly more trustworthy research conclusions, and the profit motive has obvious conflicts with "benefit to humanity" (if someone pays you to research a treatment for their disease, do you post the findings when done? Or hold them privately for the next person with the same problem?). But maybe there are other domains in which the market could support a (non-billionaire's) project for better-validated research.

Re: Statistics Done Wrong – The woefully complete guide

#20

Post by merckx »

Tycho wrote: post #17

If I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating the application of statistical methods being used; a replication division for checking that experiments do actually replicate; and a corruption division for investigating suspicious influences on the research. It would be tempting to then generate some sort of…

Something along those lines: Elizabeth Iorn's Reproducibility Initiative. [1] And an opinion piece which gives some context. [2]

[1] https://www.scienceexchange.com/reproducibility

[2] http://www.newscientist.com/article/mg21528826.000-is-medica...

Post reply on HN