Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I’m Not a Fan of R-Squared

johnmyleswhite.com

Post reply on HN
Print view View original

11–20 of 55 posts

Re: Why I’m Not a Fan of R-Squared

#11

Post by unfamiliar »

nabla9 wrote: post #4
Noseshine wrote: post #3

> “does my model perform worse than the true model?” What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning.

True model is the probability distribution that generates the observed data.

No, the "true model" he seems to be referring to is the mean of the distribution of the noisy observations. In the example, the distribution is the normal distribution and the mean is log(x), which he is referring to as the "true model".

The notation is pretty sloppy for someone handing out statistical advice.

Re: Why I’m Not a Fan of R-Squared

#12

Post by nabla9 »

unfamiliar wrote: post #11
nabla9 wrote: post #4

Earlier quoted context omitted.

True model is the probability distribution that generates the observed data.

No, the "true model" he seems to be referring to is the mean of the distribution of the noisy observations. In the example, the distribution is the normal distribution and the mean is log(x), which he is referring to as the "true model". The notation is pretty sloppy for someone handing out statistical advice.

I don't think his usage of these terms is sloppy at all.

The main point of his argument is that R² compares to observed mean and not into true model behind the data.

Terms "true model", "true distribution" and "true population" are well defined concepts in statistics – unless Jorma Rissanen is nearby.

Re: Why I’m Not a Fan of R-Squared

#13

Post by nabla9 »

DiabloD3 wrote: post #6
nabla9 wrote: post #4

Earlier quoted context omitted.

True model is the probability distribution that generates the observed data.

In other words, what was actually observed?

No. "true model", "true distribution" and "true population" is what generates the data.

Re: Why I’m Not a Fan of R-Squared

#15

Post by yiyus »

Noseshine wrote: post #3

> “does my model perform worse than the true model?” What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning.

In the example, log(x) is the true model. It is generally unknown.

Re: Why I’m Not a Fan of R-Squared

#16

Post by yiyus »

tgb wrote: post #2

Had the author offered an alternative? Namely, can E^2 be calculated in practice?

The author says that E2 is "generally unmeasurable". Indeed, the true model will be unknown in most cases.

In my opinion, it is not about using E2 or other alternative measure to R2, but about being aware of the significance and validity of R2 and not blindly assuming that a higher value always means a better fit.

Re: Why I’m Not a Fan of R-Squared

#17

Post by yiyus »

acbart wrote: post #8

This is not very accessible for people with weak statistical backgrounds.

R-Squared is the most commonly used indicator of how good a model fits some data. The author discusses why this indicator can be misleading in some cases and shows an example. There is nothing interesting for people unfamiliar with R-Squared values or model fitting.

But in fact, it is not complicated. If you feel curious, feel free to ask any specific question you have.

Re: Why I’m Not a Fan of R-Squared

#18

Post by graeham »

Interesting article and I find it current for some problems I'm working on at the moment.

I would add a few challanges. The example is a bit a of a strawman - a log(x) function has unique properties that make the Xmax-Xmin vs R^2 work like that. In real data, rarely does a single-variable 'true model' fit as well as the example either.

Context is needed as well - depending on the use of the model, a linear or quadratic fit may be sufficient even for what is clearly a log dataset. The real failing on only for small values of x, maybe 5% of the range of total values. For this case, a bilinear model could fit quite well for the lower 5%, then the existing model for the upper 95%. It depends on the application. I like this phrase:

"When deciding whether a model is useful, a high R2 can be undesirable and a low R2 can be desirable."

Too often statistics are dominated by 'cutoff' values that people apply blindly to all situations.

What do you think of robust regression methods, where obvious outliers are down-weighted?

Re: Why I’m Not a Fan of R-Squared

#19

Post by static_noise »

a_bonobo wrote: post #5
tgb wrote: post #2

Had the author offered an alternative? Namely, can E^2 be calculated in practice?

I don't think you can calculate E^2 without the "true model", which you practically never have. The code uses the "true model" too: https://github.com/johnmyleswhite/r_squared/blob/master/util... I guess the post is similar to Anscombe's quartet [1]: a warning not to blindly trust summary statistics. [1] https://en.wikipedia.org/wiki/Anscombe%27s_quartet

So the solution the author proposes is both absolutely correct and absolutely useless in practice?

Re: Why I’m Not a Fan of R-Squared

#20

Post by lottin »

static_noise wrote: post #19
a_bonobo wrote: post #5

Earlier quoted context omitted.

I don't think you can calculate E^2 without the "true model", which you practically never have. The code uses the "true model" too: https://github.com/johnmyleswhite/r_squared/blob/master/util... I guess the post is similar to Anscombe's quartet [1]: a warning not to blindly trust summary statistics. [1] https://en.wikipedia.org/wiki/Anscombe%27s_quartet

So the solution the author proposes is both absolutely correct and absolutely useless in practice?

He's not proposing a solution as far as I can tell. He's simply using the E^2 statistic in order to illustrate the problem.
Post reply on HN