Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistical Mistakes and How to Avoid Them

cs.cornell.edu

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Statistical Mistakes and How to Avoid Them

#71

Post by RA_Fisher »

fela wrote: post #50
lisper wrote: post #44

Earlier quoted context omitted.

> It's the odds of having that results due to chance, if the null hypothesis is true[0]. Yes, that's right. I don't know why you think this is at odds with what I said. In fact, I clarified this myself a few hours ago in a sibling comment: https://news.ycombinator.com/item?id=13026907

"what are the odds that the results you observed could have arisen by chance?" If you say it like this it will very easily be misinterpreted. Once your results are in there are two cases: (1) either the null hypothesis is true and you got those results due to chance, or (2) the null hypothesis is false and there was some actual effect outside of the null hypothesis that helped you get the results. Due to this it is v…

The OP is referring to Fisher's p-value rather than the more common Neyman-Pierson method that you refer to. Fisher's method doesn't have the concept of the null hypothesis. The difference is fascinating and I do believe the Fisherian method is superior if you can't easily replicate.

Re: Statistical Mistakes and How to Avoid Them

#72

Post by glangdale »

The idea that you should 'plot the error bars' ahead of, well, looking at the data seems a bit premature. As many other comments have stated, looking at the data first is critical.

It drives me up the wall: we have 1200dpi printers, retina displays, and so on, and yet somehow people feel the need to collapse everything they've done to these giant finger-painting quality bar charts. Statistical tests are well and good, but I'm amazed at the extent to which smart people will happily plug data which they have never actually seen into statistical metrics. So a mean might be derived from 9 reasonable results and a howlingly off factor-of-2 outlier, and you can dutifully plug this series into a bunch of standard tests and speak confidently about p-values.

Re: Statistical Mistakes and How to Avoid Them

#73

Post by srean »

willhslade wrote: post #64
srean wrote: post #42

Earlier quoted context omitted.

Ah I see, my apologies, I assumed Googling those keywords would be sufficient to connect the dots. The main thing that I wanted to convey is that the consequences of CLT does not come for free. It is not remotely as widely applicable as it is made out to be. CLT is also not so narrow that you need IID random variables as is often claimed. Those assumptions can be relaxed substantially. What gets in the way in obtaini…

I've seen variations of this comment in multiple threads, and have to ask: is there a good paper or textbook that spells this out?

I would say go with books on (i) heavy tailed distributions and another on (ii) stable distributions. Communities that have quickly wizened up to the deficiencies of the Gaussian assumption are (a) mathematical finance (b) statistical analysis of network packets. Following that literature might also be useful.

Re: Statistical Mistakes and How to Avoid Them

#74

Post by srean »

thanatropism wrote: post #54
srean wrote: post #26

Earlier quoted context omitted.

> If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here. The key word in there is common . There is an entire industry of statistical techniques that do not require Gaussian assumption or for that matter any parametric assumption. I strongly feel it is time to retire the Gaussian distribution from the space it occupies. Discoveri…

The Gaussian distribution is central in continuous-time models for different reasons: https://almostsure.wordpress.com/2010/04/13/levys-characteri... Basically any reasonable* stochastic continuous process is driven by a brownian motion. Also: discontinuous processes are more or less* the sum of a brownian motion and a poisson-type process. https://en.wikipedia.org/wiki/L%C3%A9vy_process#L.C3.A9vy.E2... (* Much detai…

Yes indeed, but the Brownian motion story is weaker than the CLT story. Lot more conditions required, you have to look at it the right scale, in the right way ... then stochastic processes look very much like a Brownian motion. Well, technically the bog standard CLT is a special case of this, hence has a simpler story.

Re: Statistical Mistakes and How to Avoid Them

#75

Post by wodenokoto »

imh wrote: post #65
wodenokoto wrote: post #63

Is there a good resource to learn the underpinnings of P values and T-tests? I feel like everybody says these are important, show a formula and then arguments ensue about what p=0.95 means, and nobody seems to know this.

I think any intro stats book should do the trick. As far as I know, the material in a first stats course is pretty homogeneous. I'm not a biostatistician, but I happen to like this book [0] for introductory stuff. Amazon says you can get it used for $26. [0] https://www.amazon.com/Principles-Biostatistics-CD-ROM-Marce...

I took intro to stats at a business school and switched to computational linguistics and honestly, I have only been met with the "It is something that you do" in regards to P-values.

Ill try and look at an introductory book again and see if it satisfies my curiosity.

Re: Statistical Mistakes and How to Avoid Them

#76

Post by [deleted] »

rspeer wrote: post #61
lisper wrote: post #46

Earlier quoted context omitted.

In other words: the advantage of frequentism is that it gives you better plausible deniability when you get the wrong answer. Did I get that right?

No, and I don't know where you're getting your certainty that the answer is wrong from. How about this: find me a published scientific paper with only Bayesian results in it, no frequentist statistics at all. Argue for its superiority all you want, but I don't think anyone does it. It would be seen as a stunt.

[deleted]

Re: Statistical Mistakes and How to Avoid Them

#77

Post by jcahill »

rspeer wrote: post #61
lisper wrote: post #46

Earlier quoted context omitted.

In other words: the advantage of frequentism is that it gives you better plausible deniability when you get the wrong answer. Did I get that right?

No, and I don't know where you're getting your certainty that the answer is wrong from. How about this: find me a published scientific paper with only Bayesian results in it, no frequentist statistics at all. Argue for its superiority all you want, but I don't think anyone does it. It would be seen as a stunt.

> find me a published scientific paper with only Bayesian results in it

Trivial¹, surely?

> [B]ut I don't think anyone does it. It would be seen as a stunt.

What did you mean by "scientific"?

You'd have to hunt for a narrow reading s.t. the above holds — extant counterexamples aren't limited to any one branch.

Check it out² for yourself.

Further: cogsci's bayesian adoption is rapidly accelerating.

The replication crisis is brutalizing huge swathes of psych:

• loss of confidence in NHST is becoming near-total for many

• journals are purging in turn — e.g. BASP's p-value ban³⁴

• others are overhauling stats-in-psych entirely

J Math Psych alone has two recent special issues⁵⁶ on this.

I mean:

> no frequentist stats at all

isn't even a strawman lately, let alone an absurdity.

In some fields, it's a battle-cry.

All that being said… re:

> Argue for its superiority all you want

I wouldn't even go that far. Don't give em that.

Probability interpretation fundies

• are all wrong,

• narrow minds and waste lifespans with the cultism, and

• should at least learn of the other interpretations.

Pitching probability as a 1v1 isn't merely wrong-prime⁷ — it's doublepluswrong″⁸.

It's pseudofundamentalism: fundies uphold foundations.

Flamewars predicated on ignorance of the same gotta go, no matter how fashionable they may be.

____________________________________________________________

[1] http://enwp.org/bayesian_game

[2] http://google.com/scholar?q=fully+bayesian

[3] http://doi.org/4z8 | BASP 37

[4] http://doi.org/34p | Nature re: BASP

[5] http://doi.org/btqx | J Math Psych 72

[6] http://doi.org/btqz | J Math Psych 74

[7] http://enwp.org/all_models_are_wrong

[8] http://enwp.org/not_even_wrong

Re: Statistical Mistakes and How to Avoid Them

#78

Post by fela »

RA_Fisher wrote: post #71
fela wrote: post #50

Earlier quoted context omitted.

"what are the odds that the results you observed could have arisen by chance?" If you say it like this it will very easily be misinterpreted. Once your results are in there are two cases: (1) either the null hypothesis is true and you got those results due to chance, or (2) the null hypothesis is false and there was some actual effect outside of the null hypothesis that helped you get the results. Due to this it is v…

The OP is referring to Fisher's p-value rather than the more common Neyman-Pierson method that you refer to. Fisher's method doesn't have the concept of the null hypothesis. The difference is fascinating and I do believe the Fisherian method is superior if you can't easily replicate.

The definition of p-value is the same independent of method, as far as I can tell the only real difference is that by Neyman–Pearson you just look at whether the p-value is below a threshold, and Fisher looks at p-value as "strength of evidence" valuable in itself. It's still not the probability that your result was due to chance, it's the probability that under the null hypothesis (and you will definitely need one) you would get that value (or more extreme) by chance.

Re: Statistical Mistakes and How to Avoid Them

#79

Post by BeetleB »

amelius wrote: post #5

I don't like how the article tries to push statistics on the reader. If a CS paper compares a pair of averages, then that gives certain information. If statistics can add to that, and make the results a little more precise, then that is nice. But by no means is it absolutely necessary. And statistics will not give a conclusive result either. I think that authors should use statistics when they see fit, and when it do…

Eh. Let's see how this goes.

Profession A has a mean salary 20% higher than that of Profession B.

Yet people who are in profession A are much more likely to be in poverty than in profession B.

Yet almost any time someone compares two means, they never seem to come to this conclusion - or even consider it a possibility.

Comparing two means without other details is rarely illuminating, and often leads to wrong conclusions (which are worse than no conclusions with no data).

Re: Statistical Mistakes and How to Avoid Them

#80

Post by imh »

wodenokoto wrote: post #75
imh wrote: post #65

Earlier quoted context omitted.

I think any intro stats book should do the trick. As far as I know, the material in a first stats course is pretty homogeneous. I'm not a biostatistician, but I happen to like this book [0] for introductory stuff. Amazon says you can get it used for $26. [0] https://www.amazon.com/Principles-Biostatistics-CD-ROM-Marce...

I took intro to stats at a business school and switched to computational linguistics and honestly, I have only been met with the "It is something that you do" in regards to P-values. Ill try and look at an introductory book again and see if it satisfies my curiosity.

The general motivation for a p value is that you can model what your data should look like under the assumption that your model is correct, but you can't really say what your data should look like under the assumption that your model in incorrect. There are just too many ways that it could be incorrect.

As a concrete example, I might ask you for the distribution of the mean of N samples given that they come from the standard normal distribution (mean zero, variance 1). That's easy. The sample mean, which is itself a random variable, also is normally distributed with a mean of zero and a variance of 1/N. On the other hand, if I ask you about the mean, but the only info you have is that your data isn't from a standard normal, then it could be anything! There's no objective way to say how the sample mean is distributed, given that one crappy piece of info.

The most basic thing you can do then, is to assume that your model is true and see if your data is plausible. If I have a hypothesis that I'm flipping a fair coin and I get all heads on 10 flips, I'm going to start doubting my hypothesis. The probability of all heads or all tails with a fair coin is only 1/512=0.002. P values formalize that notion. We call the hypothesis we can model our "null hypothesis", and see if we get data that makes sense with it. If your observations are some of the most unlikely ones according to your null model, let's start doubting the model. That's it.

The benefit and trouble are both that we dodged the entire question of what an alternative to our model could be, and how the data looks under those alternatives. Ignoring that incredibly important question can give rise to a weird way of thinking, and opens the door to some conceptually mind bending mistakes, but it all comes from a simple interpretation of a p value. How unlikely is your data given your null hypothesis (given the model you're trying to test). Formally, this tends to be "what is the probability that some statistic is this unlikely or worse."

Does that make any sense?

Post reply on HN