Live data from Hacker News

Statistics Done Wrong – The woefully complete guide

refsmmat.com

21–30 of 71 posts

Re: Statistics Done Wrong – The woefully complete guide

#21

Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…

Is there any way to print the entire book as a document? Do you accept donations?

Re: Statistics Done Wrong – The woefully complete guide

#22
Quite a few years ago i devised an ambitious method to achieve significance while sitting through another braindead thesis presentation (psychology):

If you are interested in the difference of a metric scaled quantity between two groups do the following:

1.) Add 4-5 plausible control variables that you do not document in advance (questionaire, sex, age...).

2.) Write a r-script that helps you do the following: Whenever you have tested a person increment your dataset with the persons result and run a:

t-test

u-test

ordinal logistic regression over some possible bucket combinations.

3.) Do this over all permutations of the control variables. Have the script ring a loud bell when significance is achieved so data collection is stopped immediately. An added bonus is that you will likely get a significant result with a small n which enables you to do a reversed power analysis.

Now you can report that your theoretical research implied a strong effect size so you choose an appropriate small n which, as expected, yielded a significant result ;)

Re: Statistics Done Wrong – The woefully complete guide

#23
post #12

Earlier quoted context omitted.

Isn't that why the name "Data Science" was invented?

Data science sure sounds sexier...ish. The irony of people who use the "damned lies and statistics" quote snidely is that the "statistics" part is not referring to the field Statistics but the plural version of the noun statistics, which of course are easily abused. The field of Statistics is all about NOT abusing statistics.

Exactly. If you're doing statistics for yourself, it's good to know the tricks so that you don't fool yourself by mistake. Many times people use statistics to support their positions, rather than to make up their minds. If done right, data science or statistics is about the latter.

Re: Statistics Done Wrong – The woefully complete guide

#24
post #17

If I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating the application of statistical methods being used; a replication division for checking that experiments do actually replicate; and a corruption division for investigating suspicious influences on the research. It would be tempting to then generate some sort of…

I think you'd get very depressed just by the statistics, let alone the reproduction. Especially if you included journals of econometrics.

Re: Statistics Done Wrong – The woefully complete guide

#26

One way to do statistics less wrong is to move from statistical testing to statistical modelling. This is what we are trying to support with BayesHive at https://bayeshive.com Other ways of doing this include JAGS ( http://mcmc-jags.sourceforge.net/ ) and Stan ( http://mc-stan.org/ ) The advantage of statistical modelling is that it makes your assumptions very explicit, and there is more of an emphasis on effect size…

BayesHive is very interesting! I couldn't find any details on pricing, though?

Re: Statistics Done Wrong – The woefully complete guide

#27
post #19
post #17

If I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating the application of statistical methods being used; a replication division for checking that experiments do actually replicate; and a corruption division for investigating suspicious influences on the research. It would be tempting to then generate some sort of…

Something like that idea has actually already been the inspiration for at least one startup: MetaMed ( http://en.wikipedia.org/wiki/MetaMed , http://nymag.com/health/bestdoctors/2013/metamed-personalize... ) does meta-level analysis of the medical literature to determine which treatments seem effective for rare conditions, taking into account the sample size, statistical methodology, funding sources, etc. of each stu…

MetaMed, as far as I know, does basically just customized literature reviews; it isn't doing anything I'd recognize as 'meta-level analysis' like the work done by the Cochrane Collaboration or using meta-analytic techniques to directly estimate the reliability of existing medical treatments or beliefs.

Re: Statistics Done Wrong – The woefully complete guide

#28
What is puzzling to me is that many of the statistical errors showing up in all the science literature are well understood. The problem is not all the junk science that is being generated but that the current tools and culture are not readily naming and shamming these awful studies. Just as we have basic standards in other fields such as GAAP in finance why can' we have an agreed upon standard for data collection and analysis of scientific data?

Re: Statistics Done Wrong – The woefully complete guide

#29
post #21

Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…

Is there any way to print the entire book as a document? Do you accept donations?

http://sphinx-doc.org/latest/builders.html

Sphinx can output Latex for printing to PDF. We'd just need the source.

BTW, excellent refresher for statistical screw-ups, I had forgotten a quarter of these (and never learned the rest.)

Re: Statistics Done Wrong – The woefully complete guide

#30

One way to do statistics less wrong is to move from statistical testing to statistical modelling. This is what we are trying to support with BayesHive at https://bayeshive.com Other ways of doing this include JAGS ( http://mcmc-jags.sourceforge.net/ ) and Stan ( http://mc-stan.org/ ) The advantage of statistical modelling is that it makes your assumptions very explicit, and there is more of an emphasis on effect size…

BayesHive is very interesting! I couldn't find any details on pricing, though?

We're thinking about it. Everything is free for the moment and we will keep a free tier for most data analysis needs.
Post reply on HN