Statistics Done Wrong – The woefully complete guide • Hacker News Board

Re: Statistics Done Wrong – The woefully complete guide

#41

Post by capnrefsmmat » Tue, Nov 05, 2013, 10:56 PM UTC

mey wrote: post #21

capnrefsmmat wrote: post #16

Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…

Is there any way to print the entire book as a document? Do you accept donations?

Not yet. I intend to get it published so you can buy a printed copy, but that hasn't happened.

I don't accept donations, but sign up with your email and I'll let you know when you can pay me for a copy.

Re: Statistics Done Wrong – The woefully complete guide

#42

Post by capnrefsmmat » Tue, Nov 05, 2013, 10:57 PM UTC

edwintorok wrote: post #31

capnrefsmmat wrote: post #16

Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…

I like how you present various ways that you can make mistakes with statistics. One thing that is missing is like a Summary/Checklist chapter that tells you what you SHOULD do, in a few common scenarios to avoid all the mistakes presented in the previous chapter. I know its not that simple, and it depends a lot on how you are testing and what you're actually trying to achieve, but a few examples wouldn't hurt. For ex…

My draft (twice as long as what's online) has a feature like this at the end of each chapter. It needs more work to be useful, but it goes a long way to making the advice in the book actionable.

Re: Statistics Done Wrong – The woefully complete guide

#43

Post by dbaupp » Tue, Nov 05, 2013, 11:00 PM UTC

daughart wrote: post #3

As a graduate student in the life sciences, I was required to take a course on ethical conduct of science. This gave me the tools to find ethical solutions to complex issues like advisor relations, plagiarism, authorship, etc. We were also taught to keep good notes and use ethical data management practices - don't throw out data, use the proper tests, etc. Unfortunately, we weren't really taught how to do statistics…

> Unfortunately, we weren't really taught how to do statistics "the right way."

Learning the right way takes a lot of work, there's a lot of ways to analyse things, each one wrong/right in different situations. (Even teaching something as "simple" as the correct interpretation of a p-value is hard.)

Re: Statistics Done Wrong – The woefully complete guide

#44

Post by capnrefsmmat » Tue, Nov 05, 2013, 11:00 PM UTC

essayist wrote: post #32

jimmar wrote: post #15

One of the many challenges in science is that there is no publication outlet for experiments that just didn't pan out. If you do an experiment and don't find statistical significance, there aren't many journals that want to publish your work. That alone helps contribute to a bias toward publishing results that might have been found by chance. If 20 independent researchers test the same hypothesis, and there is no rea…

They are working on it. There was an effort at Oxford to track perinatal trials -- started in the 1980's. It looks like it hasn't happened yet, but that various major players (PLOS, Center for Evidence-based Medicine) want to expand the brief to cover all clinical trials: http://www.alltrials.net/about/ http://www.cochrane.org/about-us/history

For many types of clinical trials, pre-registration and publication of results through ClinicalTrials.gov is required by the FDA. I think it's been five or ten years now. Unfortunately, compliance isn't great -- something like 80% of studies registered on ClinicalTrials.gov never have results published there. 20% of registered studies never have results published anywhere.

Re: Statistics Done Wrong – The woefully complete guide

#45

Post by capnrefsmmat » Tue, Nov 05, 2013, 11:02 PM UTC

Aloisius wrote: post #25

If you want to see truly egregious uses of statistics, take a look at any paper on diet or nutrition. Be prepared to be angry. At this point, if someone published a study stating that we needed to eat not to die, I'd be skeptical of it.

Re: Statistics Done Wrong – The woefully complete guide

#46

Post by jessaustin » Tue, Nov 05, 2013, 11:11 PM UTC

neuralk wrote: post #12

dragonwriter wrote: post #9

Earlier quoted context omitted.

Isn't that why the name "Data Science" was invented?

Data science sure sounds sexier...ish. The irony of people who use the "damned lies and statistics" quote snidely is that the "statistics" part is not referring to the field Statistics but the plural version of the noun statistics, which of course are easily abused. The field of Statistics is all about NOT abusing statistics.

...ish.

Yeah, it always makes me wonder when they have to put the "science" in the name. "Computation" seems so much more timeless and elegant than "computer science", for instance. It's almost like "Democratic Republic" for nations.

Re: Statistics Done Wrong – The woefully complete guide

#47

Post by anaphor » Tue, Nov 05, 2013, 11:18 PM UTC

Tycho wrote: post #17

If I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating the application of statistical methods being used; a replication division for checking that experiments do actually replicate; and a corruption division for investigating suspicious influences on the research. It would be tempting to then generate some sort of…

Fuck yes, I would love to help do something like that. I'm not a statistician though, so I'm probably not very qualified.

Re: Statistics Done Wrong – The woefully complete guide

#48

Post by daughart » Tue, Nov 05, 2013, 11:39 PM UTC

dbaupp wrote: post #43

daughart wrote: post #3

As a graduate student in the life sciences, I was required to take a course on ethical conduct of science. This gave me the tools to find ethical solutions to complex issues like advisor relations, plagiarism, authorship, etc. We were also taught to keep good notes and use ethical data management practices - don't throw out data, use the proper tests, etc. Unfortunately, we weren't really taught how to do statistics…

> Unfortunately, we weren't really taught how to do statistics "the right way." Learning the right way takes a lot of work, there's a lot of ways to analyse things, each one wrong/right in different situations. (Even teaching something as "simple" as the correct interpretation of a p-value is hard.)

I'm sure it does, but they don't have a problem assigning other required classes, such as a one-hour-a-week communication of science class. One hour a week for a year is enough to cover a lot of material.

Re: Statistics Done Wrong – The woefully complete guide

#49

Post by Alex3917 » Tue, Nov 05, 2013, 11:55 PM UTC

capnrefsmmat wrote: post #39

craigyk wrote: post #35

Earlier quoted context omitted.

As a scientist I think you are addressing a very important problem with this book. I've taken two statistics classes, one graduate level, and even I am plagued with doubt as to wether the statistics I've used have all been applied and interpreted "correctly". That said, I think the recent spate of "a majority of science publications are wrong" stories is incredible hyperbole. Is it the raw data that is wrong (fabrica…

I based that statement off of John Ioannidis's famous paper, "Why Most Published Research Findings are False." It's open-access: http://www.plosmedicine.org/article/info:doi/10.1371/journal... He's drawn some criticism for the paper, and perhaps things aren't as bad as he makes it seem, but it is true that someone has suggested most findings are false. I may tone down the Introduction slightly.

If anything, Ioannidis's paper would hugely understate the problem, because he was only looking at the percentage of papers that couldn't be replicated. But just because a result can be replicated doesn't mean that the study is actually correct. In fact, the vast majority of wrong papers are likely very replicable, since most wrong papers are the result of bad methodology (or other process-related issues) rather than fudged data.

Re: Statistics Done Wrong – The woefully complete guide

#50

Post by neoterics » Wed, Nov 06, 2013, 12:03 AM UTC

Tycho wrote: post #17

If I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating the application of statistical methods being used; a replication division for checking that experiments do actually replicate; and a corruption division for investigating suspicious influences on the research. It would be tempting to then generate some sort of…

Doesn't the Cochrane Institute already do something similar?

They perform meta analysis of studies and talk about the validity of their statistical methods. http://summaries.cochrane.org/

Read about it in the book Bad Science by Ben Goldacre.