Statistical Formulas For Programmers • Hacker News Board

Re: Statistical Formulas For Programmers

#21

Post by kevinalexbrown » Mon, May 20, 2013, 6:25 AM UTC

If you're looking for things to add big-picture-wise, it might be helpful to specify what assumptions go into various tests/methods. In my experience, this is the biggest hangup and mistake, because a) it's more difficult to understand and b) ignoring it gives the appearance of rigor even if the test used is inappropriate for the data.

I should emphasize that this is not a nitpick or even a criticism, just a feature I would love to see. It's also what I spend a large portion of my time trying to track down, so having it in a convenient location would be nice.

Re: Statistical Formulas For Programmers

#22

Post by quackerhacker » Mon, May 20, 2013, 6:31 AM UTC

Thanks for writing this Evan.

Maybe this solidifies the fact that I'm a programmer and not a statistician, but I got lost after the 2nd hyperbole in the beginning...but I like a challenge, I may have to read it multiple times until your presentation sinks in though :)

Re: Statistical Formulas For Programmers

#23

Post by scottedwards » Mon, May 20, 2013, 6:31 AM UTC

kevinalexbrown wrote: post #21

If you're looking for things to add big-picture-wise, it might be helpful to specify what assumptions go into various tests/methods. In my experience, this is the biggest hangup and mistake, because a) it's more difficult to understand and b) ignoring it gives the appearance of rigor even if the test used is inappropriate for the data. I should emphasize that this is not a nitpick or even a criticism, just a feature…

Great suggestion. I've been amazed to find out that many coders and amateur "data scientists" don't realize that testing the assumptions is an important part of conducting statistical analyses. Part of this may be due to the recent emphasis on machine learning techniques, which tend to be assumption-free (often just assuming independence of cases in the sample).

Re: Statistical Formulas For Programmers

#24

Post by durbatuluk » Mon, May 20, 2013, 7:10 AM UTC

As a programmer and statistician I should warn newcomers about the assumptions of tests (parametric tests like test t). Test t for example need same variance and normality or you'll take wrong decisions. These formulas and p-values shouldn't be used as substitutes for graphics (you should check them first and test later).

Just take care, there's a lot of problems from using statistics in wrong way. special care with small sample size and even large ones [http://scienceblogs.com/mixingmemory/2006/10/31/jeffreylindl...]

Re: Statistical Formulas For Programmers

#25

Post by salimmadjd » Mon, May 20, 2013, 7:18 AM UTC

Great blog! I wish there was a cookbook style code with eac formula. In C or even JavaScript for those of us who have forgotten a lot of math but can think in code.

Re: Statistical Formulas For Programmers

#26

Post by salimmadjd » Mon, May 20, 2013, 7:20 AM UTC

EvanMiller wrote: post #2

Uhhhh, Evan Miller here. Not sure why my name is in the submitted title, but whatever. The current selection on that page is somewhat limited, but I hope to grow it over time. The stuff at the beginning is pretty basic (e.g. standard deviation), but things get pretty gnarly by the time you get to the Kiefer equation. At some point I'll add some more references on how to implement things, e.g. find successive zeros of…

HN displays the URL's domain. Which happens to be your name

Re: Statistical Formulas For Programmers

#27

Post by deanclatworthy » Mon, May 20, 2013, 7:35 AM UTC

A nice resource, however I'd expected to see some actual programming on this page. Perhaps some sample code in a few languages would be nice. You could put tabs on a code box to switch between a python and PHP implementation for example.

Re: Statistical Formulas For Programmers

#28

Post by pseut » Mon, May 20, 2013, 7:37 AM UTC

salimmadjd wrote: post #26

EvanMiller wrote: post #2

Uhhhh, Evan Miller here. Not sure why my name is in the submitted title, but whatever. The current selection on that page is somewhat limited, but I hope to grow it over time. The stuff at the beginning is pretty basic (e.g. standard deviation), but things get pretty gnarly by the time you get to the Kiefer equation. At some point I'll add some more references on how to implement things, e.g. find successive zeros of…

HN displays the URL's domain. Which happens to be your name

The title had the name in it too originally but it seems that a moderator changed it.

Re: Statistical Formulas For Programmers

#29

Post by th0114nd » Mon, May 20, 2013, 7:43 AM UTC

The link to "how-to-read-an-unlabeled-sales-chart.html" has an extra s after unlabeled.

Re: Statistical Formulas For Programmers

#30

Post by joosters » Mon, May 20, 2013, 8:00 AM UTC

pseut wrote: post #4

The first example, "unbiased standard deviation" is mislabeled. The estimator of the variance is unbiased but the square root of an unbiased estimator is not itself unbiased. So it's not as nit-picky as it looks; it's either a brain fart or a hole in understanding (especially since the linked Wikipedia page discusses this issue)[1,2] Not to be a dick, but getting the first example wrong like that doesn't inspire conf…

This was the first time I'd ever heard about biased/unbiased sample variances, all inspired by the original web page and seeing the (N-1) in the standard deviation formula. This inspired me to go off and read about Bessel's correction in wikipedia.

Thank you, Evan Miller! Your web page is great, it taught me new stuff from the very first formula. Much appreciated.