That was an excellent read. Thank you. I'll admit I'm often reluctant to read to much in to data I deal with daily (web analytics), as I'm unsure of how to measure its significance accurately. I'm going to dive in and learn more about this.
Statistics Done Wrong – The woefully complete guide
61–70 of 71 posts
Re: Statistics Done Wrong – The woefully complete guide
#62Earlier quoted context omitted.
Data science sure sounds sexier...ish. The irony of people who use the "damned lies and statistics" quote snidely is that the "statistics" part is not referring to the field Statistics but the plural version of the noun statistics, which of course are easily abused. The field of Statistics is all about NOT abusing statistics.
...ish. Yeah, it always makes me wonder when they have to put the "science" in the name. "Computation" seems so much more timeless and elegant than "computer science", for instance. It's almost like "Democratic Republic" for nations.
It gets worse with Political Science, or perhaps the broader Social Sciences.
Re: Statistics Done Wrong – The woefully complete guide
#63Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…
As a scientist I think you are addressing a very important problem with this book. I've taken two statistics classes, one graduate level, and even I am plagued with doubt as to wether the statistics I've used have all been applied and interpreted "correctly". That said, I think the recent spate of "a majority of science publications are wrong" stories is incredible hyperbole. Is it the raw data that is wrong (fabrica…
That said, I think the recent spate of "a majority of science publications are wrong" stories is incredible hyperbole. Is it the raw data that is wrong (fabricated)?
This is only a good working assumption of some (open access) journals and of papers (co-)authored exclusively by nationals of some countries. That's a lot of papers.
The main conclusions? One or two minor side points? What if the broad strokes are right but the statistics are sloppy?
If the main conclusions are right but the statistics are sloppy the paper is true, not false.
My confidence in what Ioannidis published went up significantly on learning that epidemiology is mostly bullshit[0] and "Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility."[1]
I hope the author of the textbook does not listen to you.
[0]http://lesswrong.com/lw/72f/why_epidemiology_will_not_correc...
[1]http://blogs.nature.com/news/2011/09/reliability_of_new_drug...
Re: Statistics Done Wrong – The woefully complete guide
#64If I was a billionaire, I would set up some sort of screening lab for scientific/academic/research papers. There would be a statistics division for evaluating the application of statistical methods being used; a replication division for checking that experiments do actually replicate; and a corruption division for investigating suspicious influences on the research. It would be tempting to then generate some sort of…
SIGMOD (a database research conference) has set up a reproducibility committee [1]. Their goal is to ensure that the results can be reproduced by someone from the outside. If they succeed, you get an additional label for your graphs saying "Approved by the SIGMOD reproducibility committee."
Notably, this is easier in computer science as you don't need to wait for hundreds of patients to turn up having a certain condition.
Re: Statistics Done Wrong – The woefully complete guide
#65Earlier quoted context omitted.
You might want to take a look at Leanpub ( https://leanpub.com ) if you haven't already. Seems like it may suit you.
Unfortunately, Leanpub exclusively uses Markdown, which doesn't support things like using a BibTeX bibliography for references. I also don't think there's a way to customize the design of the print book, apart from cover images and such. I'm writing using reStructuredText and Sphinx because they give me a bibliography, an index, full-text search, cross-referencing, custom environments (e.g. boxes for examples, tips,…
this is the exact reason i'm moving away from markdown as my principle note taking tool towards restructuredText.
Re: Statistics Done Wrong – The woefully complete guide
#66Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…
I've been reading this on/off for the last day. One random UX suggestion:
Don't use black for your text, use something close (e.g. #333).
There are a million other UX tips that you can probably get from a real expert, but the black one I noticed.
Re: Statistics Done Wrong – The woefully complete guide
#67Hey everyone, I'm the author of this guide. It's come full circle -- I posted it a week ago in a "what are you working on?" Ask HN post, someone posted it to Metafilter and reddit, and it made its way to Boing Boing and Daily Kos before coming back here. I'm currently working on expanding the guide to book length, and considering options for publication (self-publishing, commercial publishers, etc.). It seems like a…
I've been reading this on/off for the last day. One random UX suggestion: Don't use black for your text, use something close (e.g. #333). There are a million other UX tips that you can probably get from a real expert, but the black one I noticed.
And another suggestion:
http://en.wikipedia.org/wiki/Simpson's_paradox
... might be nice to add to your list.
Re: Statistics Done Wrong – The woefully complete guide
#68Earlier quoted context omitted.
As a scientist I think you are addressing a very important problem with this book. I've taken two statistics classes, one graduate level, and even I am plagued with doubt as to wether the statistics I've used have all been applied and interpreted "correctly". That said, I think the recent spate of "a majority of science publications are wrong" stories is incredible hyperbole. Is it the raw data that is wrong (fabrica…
I based that statement off of John Ioannidis's famous paper, "Why Most Published Research Findings are False." It's open-access: http://www.plosmedicine.org/article/info:doi/10.1371/journal... He's drawn some criticism for the paper, and perhaps things aren't as bad as he makes it seem, but it is true that someone has suggested most findings are false. I may tone down the Introduction slightly.
There is also a big divide between statistical rigor in "science" research and "medical" research. In their defense, I think it's just extremely difficult for most medical research studies to get the kinds of random or N needed for reliable statistics.
Also regarding John Ioannidis's essay (not paper):
First, he uses the blanket term "research" in his meta-analysis (or at least examples) but his work seems focused primarily on medical research studies. Second, I'm not sure he clearly defines what it means to be "False", or for "most" published research to be "false".
Let's say there is clearly a right and a wrong answer to a question, and up until yesterday, publications A, B and C had concluded the wrong answer. But someone releases a newer, more rigorous finding D that refutes A, B and C conclusively and choses the correct answer. I wouldn't consider this particular field to be 75% wrong after the publication of D. (Though it accurately could have been described as close to 0% conclusive before D). For any particular line of inquiry, the quality of research in this area seems like it should be shifted strongly toward the maximum exemplar of this body of work, and not it's average.
Re: Statistics Done Wrong – The woefully complete guide
#69Earlier quoted context omitted.
As a scientist I think you are addressing a very important problem with this book. I've taken two statistics classes, one graduate level, and even I am plagued with doubt as to wether the statistics I've used have all been applied and interpreted "correctly". That said, I think the recent spate of "a majority of science publications are wrong" stories is incredible hyperbole. Is it the raw data that is wrong (fabrica…
"a majority of science publications are wrong" As a scientist, I think this is probably correct. In my experience, a great majority of publications draw improper statistical conclusions, and I believe many of these are wrong in substance . "What if the broad strokes are right but the statistics are sloppy?" Publishing statements as statements of truth when they are improperly or falsely backed up would be better desc…
I take the view that "a majority of science publications are wrong" is a purposefully misleading and sensationalistic take, even though it may be technically true. IMO, only the leading-edge of known science should factor into such studies, and I think that is probably not "mostly wrong". After all, even if there has only been 1 rock-solid publication in favor of a round Earth that is preceded by 99 publications in support of a flat-Earth, I would't call that field 99% wrong.
Re: Statistics Done Wrong – The woefully complete guide
#70Earlier quoted context omitted.
As a scientist I think you are addressing a very important problem with this book. I've taken two statistics classes, one graduate level, and even I am plagued with doubt as to wether the statistics I've used have all been applied and interpreted "correctly". That said, I think the recent spate of "a majority of science publications are wrong" stories is incredible hyperbole. Is it the raw data that is wrong (fabrica…
That said, I think the recent spate of "a majority of science publications are wrong" stories is incredible hyperbole. Is it the raw data that is wrong (fabricated)? This is only a good working assumption of some (open access) journals and of papers (co-)authored exclusively by nationals of some countries. That's a lot of papers. The main conclusions? One or two minor side points? What if the broad strokes are right…
To be accurate the paper should then come to the conclusion that "a majority of published work is inconclusive and fails in the proper application of statistics used to support their claims" instead of "a majority of science publications are wrong". The second is just pure sensationalist troll.
I'm not arguing against the work itself, or against more rigorous application of statistics. I'm just arguing against sensationalistic and inflammatory language. Anyone who practices science in a particular field for any length of time will have a pretty good idea of what work is "good" and "bad". Certainly they are smart enough to ignore previous work that has been refuted and/or retracted, and it's not really fair for this previous work to contribute to assessments of what % of the field is "wrong".