Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistics Done Wrong – The woefully complete guide

refsmmat.com

Post reply on HN
Print view View original

31–40 of 71 posts

Re: Statistics Done Wrong – The woefully complete guide

#31

Post by edwintorok »

capnrefsmmat wrote: post #16

Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…

I like how you present various ways that you can make mistakes with statistics.

One thing that is missing is like a Summary/Checklist chapter that tells you what you SHOULD do, in a few common scenarios to avoid all the mistakes presented in the previous chapter. I know its not that simple, and it depends a lot on how you are testing and what you're actually trying to achieve, but a few examples wouldn't hurt.

For example: I have two sets of measurements, and I want to know: * is there a statistically significant difference between them * if yes, how much is the difference

A somewhat simplistic way I'd do that is to do a two-sample t-test for question #1, and to compute statistics for the difference between the samples (mean, median, confidence intervals), but doing just that I might've already committed some of the mistakes that your site warns about, for example I completely disregarded the power of the test.

FWIW I like parts of this book on statistics which focuses on statistics in the domain of computer systems / network, although it is rather too long: http://perfeval.epfl.ch/

Re: Statistics Done Wrong – The woefully complete guide

#32

Post by essayist »

jimmar wrote: post #15

One of the many challenges in science is that there is no publication outlet for experiments that just didn't pan out. If you do an experiment and don't find statistical significance, there aren't many journals that want to publish your work. That alone helps contribute to a bias toward publishing results that might have been found by chance. If 20 independent researchers test the same hypothesis, and there is no rea…

They are working on it. There was an effort at Oxford to track perinatal trials -- started in the 1980's. It looks like it hasn't happened yet, but that various major players (PLOS, Center for Evidence-based Medicine) want to expand the brief to cover all clinical trials:

http://www.alltrials.net/about/

http://www.cochrane.org/about-us/history

Re: Statistics Done Wrong – The woefully complete guide

#33

Post by RogerL »

jimmar wrote: post #15

One of the many challenges in science is that there is no publication outlet for experiments that just didn't pan out. If you do an experiment and don't find statistical significance, there aren't many journals that want to publish your work. That alone helps contribute to a bias toward publishing results that might have been found by chance. If 20 independent researchers test the same hypothesis, and there is no rea…

This is called the "file drawer effect".

Re: Statistics Done Wrong – The woefully complete guide

#35

Post by craigyk »

capnrefsmmat wrote: post #16

Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…

As a scientist I think you are addressing a very important problem with this book. I've taken two statistics classes, one graduate level, and even I am plagued with doubt as to wether the statistics I've used have all been applied and interpreted "correctly". That said, I think the recent spate of "a majority of science publications are wrong" stories is incredible hyperbole. Is it the raw data that is wrong (fabricated)? The main conclusions? One or two minor side points? What if the broad strokes are right but the statistics are sloppy?

People also need to realize what while the Discussion and Conclusion section of publications may often read like statements of truth, they're usually just a huge lump of spinoff hypotheses in prose form. Despite my frequent frustrations with the ways science could be better, the overall arrow of progress points in the right direction. Science isn't a process where the goal is to ensure that 100% of what gets published is correct, but whereby previous assertions can be refuted and corrected.

Edit:

To be more specific, I think the statement in your Introduction is overly critical: "The problem isn’t fraud but poor statistical education – poor enough that some scientists conclude that most published research findings are probably false". I would change it to say: "conclude that most published research findings contain (significant) errors", or something along those lines.

Re: Statistics Done Wrong – The woefully complete guide

#36

Post by tokenadult »

I see the author of this interesting site is active in this thread. You may already know about this, but for onlookers I will mention that Uri Simonsohn and his colleagues

http://opim.wharton.upenn.edu/~uws/

have published a lot of interesting papers advising psychology researchers how to avoid statistical errors (and also how to detect statistical errors, up to and including fraud, by using statistical techniques on published data).

Re: Statistics Done Wrong – The woefully complete guide

#37

Post by evacuationdrill »

VladRussian2 wrote: post #34

whenever there is discussion about statistics role in science (sometimes even going as far as crossing into how science is statistics) i always remember this: http://en.wikipedia.org/wiki/Oil_drop_experiment#Fraud_alleg...

More revealing than the fraud allegations is the next section discussing the way results had pressure from his and other experiments' results for years, delaying our arrival at a more precise measurement. It reads as though it wasn't so much malice as it was self-doubt that lead to the scientists' actions.

Re: Statistics Done Wrong – The woefully complete guide

#38

Post by ronaldx »

craigyk wrote: post #35
capnrefsmmat wrote: post #16

Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…

As a scientist I think you are addressing a very important problem with this book. I've taken two statistics classes, one graduate level, and even I am plagued with doubt as to wether the statistics I've used have all been applied and interpreted "correctly". That said, I think the recent spate of "a majority of science publications are wrong" stories is incredible hyperbole. Is it the raw data that is wrong (fabrica…

"a majority of science publications are wrong"

As a scientist, I think this is probably correct. In my experience, a great majority of publications draw improper statistical conclusions, and I believe many of these are wrong in substance.

"What if the broad strokes are right but the statistics are sloppy?"

Publishing statements as statements of truth when they are improperly or falsely backed up would be better described as politics than science.

Re: Statistics Done Wrong – The woefully complete guide

#39

Post by capnrefsmmat »

craigyk wrote: post #35
capnrefsmmat wrote: post #16

Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…

As a scientist I think you are addressing a very important problem with this book. I've taken two statistics classes, one graduate level, and even I am plagued with doubt as to wether the statistics I've used have all been applied and interpreted "correctly". That said, I think the recent spate of "a majority of science publications are wrong" stories is incredible hyperbole. Is it the raw data that is wrong (fabrica…

I based that statement off of John Ioannidis's famous paper, "Why Most Published Research Findings are False." It's open-access:

http://www.plosmedicine.org/article/info:doi/10.1371/journal...

He's drawn some criticism for the paper, and perhaps things aren't as bad as he makes it seem, but it is true that someone has suggested most findings are false.

I may tone down the Introduction slightly.

Re: Statistics Done Wrong – The woefully complete guide

#40

Post by r0muald »

jimmar wrote: post #15

One of the many challenges in science is that there is no publication outlet for experiments that just didn't pan out. If you do an experiment and don't find statistical significance, there aren't many journals that want to publish your work. That alone helps contribute to a bias toward publishing results that might have been found by chance. If 20 independent researchers test the same hypothesis, and there is no rea…

There's Figshare " rel="nofollow">http://figshare.com/> to publish negative data.
Post reply on HN