Live data from Hacker News

Why I’m Not a Fan of R-Squared

johnmyleswhite.com

21–30 of 55 posts

Re: Why I’m Not a Fan of R-Squared

#21
post #18

Interesting article and I find it current for some problems I'm working on at the moment. I would add a few challanges. The example is a bit a of a strawman - a log(x) function has unique properties that make the Xmax-Xmin vs R^2 work like that. In real data, rarely does a single-variable 'true model' fit as well as the example either. Context is needed as well - depending on the use of the model, a linear or quadrat…

I'm not the author, but I'm a huge fan of robust regression. I make between $500-2000/month off a trading strategy based on such a method. (The method is basically Bayesian linear regression, but using an error model that has a heavier tail than a gaussian.)

But a really important thing when using such methods is the lucas critique. When you need to use robust regression you are definitively in a space where all the simple and generic stuff (e.g. linear regression) doesn't work. So at this point it's important to validate the underlying assumptions behind the robust regression scheme.

E.g., in my trading strategy, I've gone to great lengths to make sure the tail behavior I'm modelling is an overestimate of reality.

Re: Why I’m Not a Fan of R-Squared

#22
I find this very confusing, but I guess I'm not the intended target audience. Not that I say it's wrong, but I don't really see the point.

Do people really expect R^2 to measure the fit of the model to the true model? R^2 measures the fit of the model to the data: i.e. how well does the model perform in predicting the outcomes. In his first example is clear that all the models are equally useless: the noise dominates and the predictive power of the models is close to zero. In the second example the predictive power of all the models has improved, because there is a clear trend. The true model predicts much better than the others now, but each model predicts better than in the previous example.

In the first example, he concludes: "Even though R^2 suggests our model is not very good, E^2 tells us that our model is close to perfect over the range of x."

Actually our model is "better than perfect". The R^2 for the linear model (0.0073) and for the quadratic model (0.0084) is slightly better than for the true model (0.0064). Of course this is not a problem specific to the R^2 measure (the MSE for the linear and quadratic fits is lower than for the true generating function) and can be explained because the linear and quadratic models overfit. E^2 is essentially the ratio the 1-R^2 values (minus one). We get -0.00083 and -0.00193 for the linear and quadratic models respectively (the ratios before substracting one are 0.9992 and 0.9981).

In the second example,"visual inspection makes it clear that the linear model and quadratic models are both systematically inaccurate, but their values of R^2 have gone up substantially: R^2=0.760 for the linear model and R^2=0.997 for the true model. In contrast, E^2=85.582 for the linear model, indicating that this data set provides substantial evidence that the linear model is worse than the true model."

The R^2 already indicates that the linear model (R^2=0.760) and the quadratic model (R^2=0.898) are worse than the true model (R^2=0.997). The fractions of unexplained variance are 0.240, 0.102 and 0.003 respecively and it's clear that the last one performs much better than the others before we take the ratios and substract one to calculate the E^2 values 85.6 and 35.7 for the linear and quadratic models respectively.

(By the way: "we’ll work with an alternative R^2 calculation that ignores corrections for the number of regressors in a model." That's not an alternative R^2, that's the standard R^2. The adjusted R^2 that takes into account the number of regressors is the alternative one.)

Re: Why I’m Not a Fan of R-Squared

#23
post #2

Had the author offered an alternative? Namely, can E^2 be calculated in practice?

My main issue with R^2 is that it is an artificial indicator. It has no business meaning. The best alternative is to build a metric that is related to your business issue. How many $ do you loss if you're wrong by 1%? Or by 10 units? Do your business loss is linear in your error? Is it symmetrical?

These are the kind of questions you have to ask yourself. Defining a metric is hard, and there is no good shortcut.

Re: Why I’m Not a Fan of R-Squared

#25
post #4

Earlier quoted context omitted.

True model is the probability distribution that generates the observed data.

No, the "true model" he seems to be referring to is the mean of the distribution of the noisy observations. In the example, the distribution is the normal distribution and the mean is log(x), which he is referring to as the "true model". The notation is pretty sloppy for someone handing out statistical advice.

The true model is

  y = log(x) + u
Where u is an error/noise term, and in this case takes a normal distribution. The error term is meant to Hoover up all the stuff you can't account for because it's impossible to measure everything. As ever, what you're trying to estimate is the parameters for everything but the error term.

He's maybe being a little sloppy with the notation, but TBH it's a pedantic distinction that I haven't seen anyone make a big deal of since I last took intro stats. In R, for example, you'd specify the model as "y ~ log(x)" and leave it at that.

Re: Why I’m Not a Fan of R-Squared

#26
post #13
post #6

Earlier quoted context omitted.

In other words, what was actually observed?

No. "true model", "true distribution" and "true population" is what generates the data.

Which is a misnomer, because the data probably isn't generated by a model.

Re: Why I’m Not a Fan of R-Squared

#28
post #27

This is a weird critique of R-Squared. You (should) learn very early on that comparing R-Squared values of different models is a no-no.

And you're claiming it's weird because people learn this early on? Maybe that's where we disagree -- it's not as common to learn about the issues with R^2 as you might think.

Re: Why I’m Not a Fan of R-Squared

#29

> “does my model perform worse than the true model?” What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning.

It's a bit of a philosophical thing. The data arose because of results of some large set of processes, all unobserved. Assuming a deterministic world—just for convenience of argument here—there's no actual underlying distribution which caused the data. Just a big equation with too many unknowns.

In practice, though, these systems are often well-described by distributions. In common parlance for statistics, people thus sometimes like to talk about that large set of unknowns as though it really were just some arbitrarily complex distribution which we don't know: the true model.

From a frequentist perspective, the idea of the true model is very important. Many arguments arise by considering it and trying to understand how to reduce the distance between your estimators and the unknown truth.

Bayesian foundations avoid this entirely, refusing to posit an unknown and instead talking about refinements of what we know.

Re: Why I’m Not a Fan of R-Squared

#30
I'm a huge fan of R^2 and you should be too.

The simple way to think about R^2 is that it is a measure of relative predictive accuracy (and that is exactly how it is calculated). This is both a more accurate (and a more useful definition for most tasks the HN crowd would work on) than saying R^2 is a measure of the distance from the true model.

All the figures and findings in the post are completely reasonable given that definition (the linear model performing better in the first case is due to the high variance of the author's data, other samplings would lead to the reverse result).

Post reply on HN