Why I’m Not a Fan of R-Squared
johnmyleswhite.com
Why I’m Not a Fan of R-Squared
1–10 of 55 posts
Re: Why I’m Not a Fan of R-Squared
#2Had the author offered an alternative? Namely, can E^2 be calculated in practice?
Re: Why I’m Not a Fan of R-Squared
#3 > “does my model perform worse than the true model?”
Re: Why I’m Not a Fan of R-Squared
#4> “does my model perform worse than the true model?” What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning.
True model is the probability distribution that generates the observed data.
Re: Why I’m Not a Fan of R-Squared
#5Had the author offered an alternative? Namely, can E^2 be calculated in practice?
I don't think you can calculate E^2 without the "true model", which you practically never have. The code uses the "true model" too: https://github.com/johnmyleswhite/r_squared/blob/master/util...
I guess the post is similar to Anscombe's quartet [1]: a warning not to blindly trust summary statistics.
Re: Why I’m Not a Fan of R-Squared
#6> “does my model perform worse than the true model?” What is "true model"? I can't make head nor tail of that term. I've never heard this before, nor does it make sense to me when I take just the word meaning.
True model is the probability distribution that generates the observed data.
In other words, what was actually observed?
Re: Why I’m Not a Fan of R-Squared
#7Re: Why I’m Not a Fan of R-Squared
#8This is not very accessible for people with weak statistical backgrounds.
Re: Why I’m Not a Fan of R-Squared
#9Earlier quoted context omitted.
True model is the probability distribution that generates the observed data.
In other words, what was actually observed?
Not really. What is observed is data, not a model/distribution. One can fit a given model to this data, but they do not, on their own, tell you which model generated them.
Re: Why I’m Not a Fan of R-Squared
#10Earlier quoted context omitted.
True model is the probability distribution that generates the observed data.
In other words, what was actually observed?
Maybe you've gotten it but I don't know. Here is a maybe over-tired coin-flipping example:
Say you have a coin that might be unfair, and you want to estimate its bias. You flip it a bunch of times, and it mostly lands heads.
Predictions of the coin's bias based on this observed data are usually going to be that it's biased some % towards heads. (unless maybe you have a strong prior, but that's a different topic)
But there is also a chance that really the coin is fair, or even biased towards tails and you just got unlucky in your flips.
There's a mismatch between the true model (the coins actual bias) and the observed data (the result of your flips) because of this chance of `unluckiness'