Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I’m Not a Fan of R-Squared

johnmyleswhite.com

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Why I’m Not a Fan of R-Squared

#41

Post by tomrod »

jsprogrammer wrote: post #39
pessimizer wrote: post #36

Earlier quoted context omitted.

Unless you have data, that is. The data is the basis for assuming that a process has generated data. Either that, or the data has existed for all eternity, and therefore could never have been collected.

"data" is your own perception, however. Can you give an example of a data generating process?

We can create them!

Suppose I take the function y = log(x) and add random white noise. The function log() and the parameters on the random white noise process are the data generating process. We could then fit a model y = \beta X + \epsilon, and then compare the "true" (first) model to our second model. When the natural world generates our data, the idea behind all this is the same: there is a process which generates the data, and the data reveals information about that process to an approximate degree.

Some reads:

[1] http://www.rimini.unibo.it/fanelli/econometric_models2_2012....

[2] https://en.wikipedia.org/wiki/Data_generating_process

Re: Why I’m Not a Fan of R-Squared

#42

Post by [deleted] »

scottfr wrote: post #30

I'm a huge fan of R^2 and you should be too. The simple way to think about R^2 is that it is a measure of relative predictive accuracy (and that is exactly how it is calculated). This is both a more accurate (and a more useful definition for most tasks the HN crowd would work on) than saying R^2 is a measure of the distance from the true model. All the figures and findings in the post are completely reasonable given…

[deleted]

Re: Why I’m Not a Fan of R-Squared

#43

Post by jsprogrammer »

tomrod wrote: post #41
jsprogrammer wrote: post #39

Earlier quoted context omitted.

"data" is your own perception, however. Can you give an example of a data generating process?

We can create them! Suppose I take the function y = log(x) and add random white noise. The function log() and the parameters on the random white noise process are the data generating process. We could then fit a model y = \beta X + \epsilon, and then compare the "true" (first) model to our second model. When the natural world generates our data, the idea behind all this is the same: there is a process which generates…

Can you give a non-synthetic, ie. natural, example data generation process?

Edit: I don't know if people don't like the grammar, or what?

How about this:

Can you give a non-synthetic, ie. natural, example of a data generating process?

Re: Why I’m Not a Fan of R-Squared

#44

Post by obastani »

tel wrote: post #29
Noseshine wrote: post #3

> “does my model perform worse than the true model?” What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning.

It's a bit of a philosophical thing. The data arose because of results of some large set of processes, all unobserved. Assuming a deterministic world—just for convenience of argument here—there's no actual underlying distribution which caused the data. Just a big equation with too many unknowns. In practice, though, these systems are often well-described by distributions. In common parlance for statistics, people thu…

It's not just philosophical -- there are examples of practical situations where there is a true model (especially in more traditional applications of statistics):

- Polling for presidential election: The "true model" is the voting preferences of all 300 million Americans. A (uniformly) random sample of N Americans can be used to estimate the true model roughly with standard error 1/sqrt(N).

- Particle physics: The "true model" is the decay probabilities of various particles as computed by quantum mechanics. Different models (and parameters) yield different decay probabilities, and experiments can be used to choose between models and/or estimate model parameters.

Of course, oftentimes the true model is philosophical as you describe.

Re: Why I’m Not a Fan of R-Squared

#45

Post by sixbrx »

An interesting tangential argument I had with a colleague recently was how to actually calculate R^2 when testing model predictions on new data (emphasis on new data).

He claimed the conventional way is as the square of the correlation coefficient of the two lists of data, whereas I was suggesting 1 - SSE/SST in accord with references I read (well, Wikipedia), which latter method yields lower values for his data. I know they are equal on training data when using linear regression with constant term, but generally they differ. Correlation way being scale and shift invariant to me disqualifies it as a measure for predicting new data.

Unfortunately he was able to barrage me with many online references that just say that "R^2 is the square of the correlation coefficient", which without very careful reading of the context (fitting not predicting), and sometimes even with such careful reading, makes his interpretation look correct. I find the whole thing rather exasperating...

It also occurs to me that just as a question of convention, that I may be wrong which wouldn't surprise me as I'm new to modeling.

So: Do most modelers report correlation^2 as their R^2 values for holdout tests? I wonder have other modelers here encountered this confusion?

Re: Why I’m Not a Fan of R-Squared

#47

Post by mattb314 »

"As such, R^2 answers the question: 'does my model perform better than a constant model?' But we often would like to answer a very different question: 'does my model perform worse than the true model?' "

Maybe I'm over-generalizing here, but I think this fundamental assumption is untrue for most people who use statistical models to solve actual problems. All of the "undesirable" behavior of the R^2 metric makes complete sense when you view it as a comparison with the most naive model (the constant model), and while R^2 certainly doesn't measure how close to "true" a model is, it very well captures the utility of using a more sophisticated model over an extremely simple one, which (I believe) is a critical question.

For example, if you had predict a process where measurement noise overwhelms variation in the process itself, as in his first example of log(x) from .99 to 1.0, then the constant model is pretty much the best you can do, and both log(x) itself and the linear model offer little additional benefit. Getting low R^2 values for those two models makes total sense--they offer no marginal benefit. In fact, if you had to make the decision "should I use log(x) or the constant model?", you're often better off going with the constant model for simplicity and predictability (unless you have domain knowledge motivating a different choice).

I like well thought-out articles like this one on statistical concepts because they make me think about things I often take for granted, and while treating R^2 like a better version of RMSE or MAD is clearly wrong, it often better captures things people actually care about by taking the difficulty of the problem into account. If you're doing advanced statistical modeling, it's easy to underestimate how common it is for a beginner to celebrate getting (say) 96% classification accuracy on a 2-class problem where one class makes up more than 95% of the samples--an issue that using R^2 can quickly reveal.

tldr: Awesome article but (imo) R^2 is more useful than most other metrics, not less

Re: Why I’m Not a Fan of R-Squared

#48

Post by kgwgk »

kgwgk wrote: post #22

I find this very confusing, but I guess I'm not the intended target audience. Not that I say it's wrong, but I don't really see the point. Do people really expect R^2 to measure the fit of the model to the true model? R^2 measures the fit of the model to the data: i.e. how well does the model perform in predicting the outcomes. In his first example is clear that all the models are equally useless: the noise dominates…

There is now a new definition for E^2 (one minus the ratio of the R^2 of the model and the "true" model) which doesn't solve the most obvious issue: getting negative values for a measure called "something squared". The values of E^2 in the first example are now -0.13 for the linear model and -0.30 for the quadratic model. In the second example, they are 0.24 and 0.10 respectively.

The graphical representation is a bit misleading. Leaving aside the fact that in the first example MSE_T is between MSE_M and MSE_C, this drawing make E^2 and R^2 seem more complementary than they really are. E^2 is the length of the blue bar as a fraction of the total length (blue+orange). R^2, however, is the length of the orange bar as a fraction of the distance from the end of the bar to the origin (not shown in the chart).

Edit: there is a new addition to the post, re-expressing E^2 in terms of a mean/variance decomposition. It should be kept in mind that the derivation presented is only asymptotically correct. In a small sample, the cross term does not vanish and the variance of the observations around the "true" value is not exactly sigma^2. In the second example, E^2 calculated using this new definition is quite similar (0.2373 and 0.0991 for the linear and quadratic models, compared to the previous values of 0.2382 and 0.0994). In the first example, however, the values we get from the new definition are far from the previous values: 0.0646 vs -0.129 for the linear model, 0.1528 vs -0.297 for the quadratic model.

Edit2: changed "approximation" to "new definition", "good" to "similar" and "exact" to "previous" in the previous paragraph. I'm not sure if he was suggesting to use this formula to calculate E^2 instead of the previous one. Anyway, it doesn't matter because this is not something that can be calculated at all unless the "true model" is known.

Re: Why I’m Not a Fan of R-Squared

#49

Post by mrandrewandrade »

robzyb wrote: post #40
mrandrewandrade wrote: post #35

Earlier quoted context omitted.

I think what John is saying is that people commonly use R^2 as measure of model fit where something like root mean squared error (RMSE) gives a better measure of model fit (by measuring the distance from the true model) depending on the model. Just using R^2 blindly for most tasks you would work on can lead to choosing an incorrect model. I think the main take-away from the post is to better understand the correct me…

> I think what John is saying is that people commonly use R^2 as measure of model fit where something like root mean squared error (RMSE) gives a better measure of model fit (by measuring the distance from the true model) I don't mean to be rude, but that is definitely not what he is saying. There are two important things I'd like to clarify: - It is wrong to call the alternative measure "the RMSE". The alternative t…

I think we are saying the same thing in different words, and I might be confusing "an alternative" with "a comparison". John compares R^2 with E^2, but RMSE can be considered an alternative to using R^2 in certain cases.

If you go back to the first line: > People sometimes use R^2 as their preferred measure of model fit.

I think the post is going over why R^2 is not recommended as 2 is not only a measure of the error, but it includes a comparison with a constant model. John defines E^2 as a comparison metric which measures how much worse the errors are than if you used the true model.

Going back to a metric for determining model fit, RMSE/MSE/MAD are all alternative measures of model fit and are useful depending on the dataset.

Re: Why I’m Not a Fan of R-Squared

#50

Post by tel »

obastani wrote: post #44
tel wrote: post #29

Earlier quoted context omitted.

It's a bit of a philosophical thing. The data arose because of results of some large set of processes, all unobserved. Assuming a deterministic world—just for convenience of argument here—there's no actual underlying distribution which caused the data. Just a big equation with too many unknowns. In practice, though, these systems are often well-described by distributions. In common parlance for statistics, people thu…

It's not just philosophical -- there are examples of practical situations where there is a true model (especially in more traditional applications of statistics): - Polling for presidential election: The "true model" is the voting preferences of all 300 million Americans. A (uniformly) random sample of N Americans can be used to estimate the true model roughly with standard error 1/sqrt(N). - Particle physics: The "t…

That's reasonably fair, but it's worth noting that even in these situations the true circumstance might be a little more nuanced than the idea of a "true model" suggests.

- In polling, it's a bit of an ideal world idea to think that the true voting preferences of all Americans are (1) fixed, (2) consistently measurable, or (3) even relevant given a lot of people won't vote. These are all sources of unknowns and variances which make me pretty unhappy with the idea of the idea of a true model even here.

- In physics there's definitely a notion of a true model taken under the assumption that one set of equations and models is "correct" (which is its own philosophical problem!) but even then experiments don't measure this true model perfectly. They're also based on correction for variance in measurement and tooling which is assumed to be eventually ignorable. In any case, this has a much more definite notion of a true model, but I still find it difficult to swallow all together.

Post reply on HN