Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistical Formulas For Programmers

evanmiller.org

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Statistical Formulas For Programmers

#41

Post by bearmf »

scottedwards wrote: post #20

Great effort, and I certainly hope more coders will get into statistics (most I know are only interested in machine learning). However, I think your definition of 1.3 "Confidence Interval around the Mean" could be improved. You state: "A confidence interval reflects the set of statistical hypotheses that won't be rejected at a given significance level. So the confidence interval around the mean reflects all possible…

Confidence intervals are inherently confusing. I have yet to hear a definition that is both correct and easily understood and remembered.

Re: Statistical Formulas For Programmers

#43

Post by Ixiaus »

pornel wrote: post #34

I hoped for it to be more "for programmers", like this one: http://gdr.geekhood.net/gdrwpl/metnum.php For me formulas written in pseudocode are much easier to understand than classic mathematical notation. For example I've learned bayesian classification, chi-square, etc. from Practical Common Lisp ( http://www.gigamonkeys.com/book/practical-a-spam-filter.html ) after failing to understand how to apply formulas from…

This is off-topic; but I couldn't help notice your user name. My RL given name is "Parnell" and people often mis-pronounce it as "Pornell"...

Re: Statistical Formulas For Programmers

#44

Post by scottfr »

kevinalexbrown wrote: post #21

If you're looking for things to add big-picture-wise, it might be helpful to specify what assumptions go into various tests/methods. In my experience, this is the biggest hangup and mistake, because a) it's more difficult to understand and b) ignoring it gives the appearance of rigor even if the test used is inappropriate for the data. I should emphasize that this is not a nitpick or even a criticism, just a feature…

Just as a rule of thumb, in general the most important assumption that is violated is independence. If the data is not independent and is instead positively correlated (the most likely form of correlation in practice) most equations will overestimate statistical significance.

A small amount of correlation can lead to very biased estimates of significance.

Correlation between observations will generally appear in any time series data or data that is arranged spatially.

Again, just a rule of thumb, but you should be very wary of the lack of independence between observations.

Re: Statistical Formulas For Programmers

#45

Post by platz »

SagelyGuru wrote: post #33

Nice but little knowledge is a dangerous thing. It is probably safer and more effective for non-statistician "data scientists" to use Robust Statistics: https://en.wikipedia.org/wiki/Robust_statistics

The wikipedia article talks much about dealing with outliers. How can outliers be removed/replaced or handled differently as the article suggests? Are the outliers not part of the data, after all? It seems like the goal of 'improving performance' here involves tweaking the data to get the results you want. What have I misunderstood here?

Re: Statistical Formulas For Programmers

#46

Post by buzzkillr2 »

timr wrote: post #36
kevinalexbrown wrote: post #21

If you're looking for things to add big-picture-wise, it might be helpful to specify what assumptions go into various tests/methods. In my experience, this is the biggest hangup and mistake, because a) it's more difficult to understand and b) ignoring it gives the appearance of rigor even if the test used is inappropriate for the data. I should emphasize that this is not a nitpick or even a criticism, just a feature…

This guy has written up many of the most common statistical tests, their interpretations, and their assumptions in a very human-readable way: http://udel.edu/~mcdonald/statintro.html I point a lot of newbs to pages on that site so that they can develop a better intuition for the methods.

This is an amazing resource thank you.

I wish I had this last semester during my statistics course.

Re: Statistical Formulas For Programmers

#47

Post by timr »

buzzkillr2 wrote: post #46
timr wrote: post #36

Earlier quoted context omitted.

This guy has written up many of the most common statistical tests, their interpretations, and their assumptions in a very human-readable way: http://udel.edu/~mcdonald/statintro.html I point a lot of newbs to pages on that site so that they can develop a better intuition for the methods.

This is an amazing resource thank you. I wish I had this last semester during my statistics course.

If you're looking for more of this, but in greater depth, pick up a copy of Biometry.

It's written for biologists, but you don't really need to know much biology to work through the examples, and the focus is inherently practical.

Re: Statistical Formulas For Programmers

#48

Post by dreen »

I think if trying to explain things like these to programmers, maybe you should consider using actual code (or even pseudo code) for this? It would have an added benefit of not requiring to know math-english (for non-native english speakers like me) or any advanced concepts of math at all.

Re: Statistical Formulas For Programmers

#49

Post by bjhoops1 »

pseut wrote: post #4

The first example, "unbiased standard deviation" is mislabeled. The estimator of the variance is unbiased but the square root of an unbiased estimator is not itself unbiased. So it's not as nit-picky as it looks; it's either a brain fart or a hole in understanding (especially since the linked Wikipedia page discusses this issue)[1,2] Not to be a dick, but getting the first example wrong like that doesn't inspire conf…

I always make a point of reading comments that start with "Not to be a dick..." :P

Re: Statistical Formulas For Programmers

#50

Post by gtani »

scottedwards wrote: post #23
kevinalexbrown wrote: post #21

If you're looking for things to add big-picture-wise, it might be helpful to specify what assumptions go into various tests/methods. In my experience, this is the biggest hangup and mistake, because a) it's more difficult to understand and b) ignoring it gives the appearance of rigor even if the test used is inappropriate for the data. I should emphasize that this is not a nitpick or even a criticism, just a feature…

Great suggestion. I've been amazed to find out that many coders and amateur "data scientists" don't realize that testing the assumptions is an important part of conducting statistical analyses. Part of this may be due to the recent emphasis on machine learning techniques, which tend to be assumption-free (often just assuming independence of cases in the sample).

    machine learning techniques, which tend to be assumption-free
ML should be a rigorous exercise in Bayesian and classical/frequentist stats, computational methods, dataset integrity, visualization etc, if you've been thru the texts by Murphy or Bishop. It often happens that people a couple years out of their last stats class only retain that high R-squared, p-, t- and f-values are what they're looking for, and heteroskedasticity and sphericity are just big words.

My evidence that ML is a rigorous exercise: the free texts listed (Barber, Mackay and Smola's are excellent, ESL not as accessible)

http://metaoptimize.com/qa/questions/186/good-freely-availab...

Post reply on HN