Nice but little knowledge is a dangerous thing. It is probably safer and more effective for non-statistician "data scientists" to use Robust Statistics: https://en.wikipedia.org/wiki/Robust_statistics
Mere 'formulas' without theory are dangerous.
Statistical Formulas For Programmers
51–60 of 66 posts
Re: Statistical Formulas For Programmers
#52Earlier quoted context omitted.
Great suggestion. I've been amazed to find out that many coders and amateur "data scientists" don't realize that testing the assumptions is an important part of conducting statistical analyses. Part of this may be due to the recent emphasis on machine learning techniques, which tend to be assumption-free (often just assuming independence of cases in the sample).
machine learning techniques, which tend to be assumption-free ML should be a rigorous exercise in Bayesian and classical/frequentist stats, computational methods, dataset integrity, visualization etc, if you've been thru the texts by Murphy or Bishop. It often happens that people a couple years out of their last stats class only retain that high R-squared, p-, t- and f-values are what they're looking for, and heteros…
Thanks, @gtani, great resource. Yeah didn't mean to imply that ML techniques are free of ANY assumptions, just that several of the popular ones like logistic regression don't have distributional assumptions. (actually, I really want to understand the VC Inequality at some point, as it seems to allow us to make conclusions about out of sample error rates without depending on distributional assumptions)
Re: Statistical Formulas For Programmers
#53I've wondered for a long time if there's a way to condense certain statistical (and probability) information into a single How-Much-Should-I-Care number. Can someone shed some light on this?
To pick a couple examples from health news in the popular press:
NB: I'm making up all the numbers here for the sake of example.
(1) A study shows that people who consume more than 10g of added salt a day live shorter lives.
But how much shorter? If it's 30 minutes shorter, I don't care about the study and I'm not going to change my behavior. If it's 6 months longer, then I'm interested and might very well do something.
(2) A study shows that people who drink 2 or more cups of coffee a day have lower risk of Alzheimer's Disease.
But how much lower risk? If the average lifetime risk is 1 in 50, and drinking coffee lowers it to 1 in 49.997, then I don't want to waste time even reading the article. If it lowers it to 1 in a 1000, then yes, I might change my behavior.
So, in the above examples, is there any way to reduce the information into a single How-Much-Should-I-Care number?
Like this:
(1) A study shows that people who consume more than 10g of added salt a day have an ____x____ factor shorter life.
(2) A study shows that people who drink 2 or more cups of coffee a day have a ____y____ factor lower risk of Alzheimer's Disease.
Then, by looking at x and y, I can tell at a glance whether some result is irrelevant, trivial, useful, or groundbreaking. I understand that it'll still be subjective in the end -- like whether $1, $10, $1000, or $10,000,000 seems like a lot of money to an individual -- but at least it'll be one number.
Re: Statistical Formulas For Programmers
#54I've wondered for a long time if there's a way to condense certain statistical (and probability) information into a single How-Much-Should-I-Care number. Can someone shed some light on this? To pick a couple examples from health news in the popular press: NB: I'm making up all the numbers here for the sake of example. (1) A study shows that people who consume more than 10g of added salt a day live shorter lives. But…
See https://en.wikipedia.org/wiki/Odds_ratio for one answer. Another "single number" answer is generated by the class of tests that determine the statistical significance of an observation; see https://en.wikipedia.org/wiki/Alpha_level for more about this.
There are many other ways to accomplish what you're talking about. The biggest problem with your made-up examples is that they are just cases of "bad reporting."
Re: Statistical Formulas For Programmers
#55I like the spirit of this article, but I wish it put much more emphasis on explaining why/when you'd need this. Examples would be very helpful.
Re: Statistical Formulas For Programmers
#56Hey Evan, from one statistics guy to another, thanks for fighting the good fight :). The formulas might benefit from examples, especially with some of the more complicated cases (KS test and onwards). The important part of statistics comes from knowing _when_ to apply something, rather than _how_ to (that part is just math/numerical analysis). A mention of the assumptions of each of these intervals would be good, too…
While you guys are here, can you recommend a good intro book for statistics?
Check out the Intro to Stats course on Udacity:
https://www.udacity.com/course/st101
Just took it earlier this year. It was informative and I enjoyed the class.
Re: Statistical Formulas For Programmers
#57Nice but little knowledge is a dangerous thing. It is probably safer and more effective for non-statistician "data scientists" to use Robust Statistics: https://en.wikipedia.org/wiki/Robust_statistics
The wikipedia article talks much about dealing with outliers. How can outliers be removed/replaced or handled differently as the article suggests? Are the outliers not part of the data, after all? It seems like the goal of 'improving performance' here involves tweaking the data to get the results you want. What have I misunderstood here?
The main difference between classical regression using ordinary least squares (OLS) and robust regression using iterative re-weighted least squares (IRLS) is that with OLS, all observations are given equal weight and with IRLS, observations may or may not be given equal weight. Essentially, IRLS gives outliers and/or influential [1] data points less weight, which may improve the performance of the overall model since these outliers/influential data would otherwise cause assumption violations using classical regression. If there are no outliers, then results from robust and classical regression converge.
I would disagree with SagelyGuru in recommending robust regression for non-statisticians, though I can see where he or she is coming from. With robust regression, you don't have to worry as much about assumptions as with classical regression. But with robust regression, you need to be aware that the underlying analytical method is different and what that means. For example, the standard robust regression implementation in R (i.e, the rlm function in the MASS package) doesn't produce t-statistics or p-values. There're also warnings that especially at lower sample sizes, the standard errors produced by rlm may be unreliable. One recommended way to obtain those p-values would be to get bootstrapped standard error estimates, so that normal-theory approximation would apply.
[1] There are different types of robust estimators (e.g., M, S, MM, etc.) that have different robustness properties.
Re: Statistical Formulas For Programmers
#58If you're looking for things to add big-picture-wise, it might be helpful to specify what assumptions go into various tests/methods. In my experience, this is the biggest hangup and mistake, because a) it's more difficult to understand and b) ignoring it gives the appearance of rigor even if the test used is inappropriate for the data. I should emphasize that this is not a nitpick or even a criticism, just a feature…
Just as a rule of thumb, in general the most important assumption that is violated is independence. If the data is not independent and is instead positively correlated (the most likely form of correlation in practice) most equations will overestimate statistical significance. A small amount of correlation can lead to very biased estimates of significance. Correlation between observations will generally appear in any…
To add to this comment, large correlation between (or among) variables will often inflate estimates of standard error, leading to underestimates of statistical significance.
Re: Statistical Formulas For Programmers
#59Earlier quoted context omitted.
The wikipedia article talks much about dealing with outliers. How can outliers be removed/replaced or handled differently as the article suggests? Are the outliers not part of the data, after all? It seems like the goal of 'improving performance' here involves tweaking the data to get the results you want. What have I misunderstood here?
The main difference between classical regression using ordinary least squares (OLS) and robust regression using iterative re-weighted least squares (IRLS) is that with OLS, all observations are given equal weight and with IRLS, observations may or may not be given equal weight. Essentially, IRLS gives outliers and/or influential [1] data points less weight, which may improve the performance of the overall model since…
Thanks for the explanation - it was helpful
Re: Statistical Formulas For Programmers
#60I've wondered for a long time if there's a way to condense certain statistical (and probability) information into a single How-Much-Should-I-Care number. Can someone shed some light on this? To pick a couple examples from health news in the popular press: NB: I'm making up all the numbers here for the sake of example. (1) A study shows that people who consume more than 10g of added salt a day live shorter lives. But…
Just a factor is often not enough. If a study would show that cell phone usage increased the risk of a certain cancer with 40%, that might still not be interesting if it's an extremely rare cancer and they found 7 cases instead of 5.
Or it doesn't show whether various correlation factors matter, or whether this is a statistical paradox. Did you know that babies of smokers are healthier than babies of non-smokers of the same weight? This is because baby born to the smoker will have decreased weight because of the smoking, whereas if the baby of the non-smoker is underweight it will be for other reasons that are often worse.
There's heaps of these kind of paradoxes and pitfalls that need to be taken in account.
What we need in newspapers and other media is a simplified abstract of the paper and an explanation or approval of a real life statistician, with no relation to the study.