31–40 of 55 posts
Earlier quoted context omitted.
No. "true model", "true distribution" and "true population" is what generates the data.
Which is a misnomer, because the data probably isn't generated by a model.
This is a weird critique of R-Squared. You (should) learn very early on that comparing R-Squared values of different models is a no-no.
And you're claiming it's weird because people learn this early on? Maybe that's where we disagree -- it's not as common to learn about the issues with R^2 as you might think.
https://en.wikipedia.org/wiki/Coefficient_of_determination#I...
Also, I find that the OP is actually more confusing on this topic than Wikipedia.
Not sure how else I can support that this is a basic fact about this metric. I'm not in the mood to find quotes in intro textbooks, etc.
Earlier quoted context omitted.
Which is a misnomer, because the data probably isn't generated by a model.
Semantics at this point. Data-generating process is a term that's also used. A model seeks to mimic or match the real process to a reasonable approximation. Hence "true model" as a scoring engine or data relationship that represents reality completely.
Semantics is the essence of communication.
Learn about information theory. It is better to calculate the cross-entropy error or KL-divergence between the data and the model.
I'm a huge fan of R^2 and you should be too. The simple way to think about R^2 is that it is a measure of relative predictive accuracy (and that is exactly how it is calculated). This is both a more accurate (and a more useful definition for most tasks the HN crowd would work on) than saying R^2 is a measure of the distance from the true model. All the figures and findings in the post are completely reasonable given…
I think the main take-away from the post is to better understand the correct measure for model fit for your specific data. For example, if you are forecasting a time series with stationary demand, mean average deviation might be the best measure of model fit, but it the case where there is seasonality with a trend, the RMSE would be a better measure [1].
Earlier quoted context omitted.
Semantics at this point. Data-generating process is a term that's also used. A model seeks to mimic or match the real process to a reasonable approximation. Hence "true model" as a scoring engine or data relationship that represents reality completely.
Except that we can't access reality completely. Assuming that there is a "true model" generating data is just that: an assumption with no real basis. Semantics is the essence of communication.
I find this very confusing, but I guess I'm not the intended target audience. Not that I say it's wrong, but I don't really see the point. Do people really expect R^2 to measure the fit of the model to the true model? R^2 measures the fit of the model to the data: i.e. how well does the model perform in predicting the outcomes. In his first example is clear that all the models are equally useless: the noise dominates…
For fits that are almost entirely close to the mean (no slope) I would expect to be saved by the F-test, but we're not here since there's a region where a linear fit fits the data at least somewhat well.
I find this very confusing, but I guess I'm not the intended target audience. Not that I say it's wrong, but I don't really see the point. Do people really expect R^2 to measure the fit of the model to the true model? R^2 measures the fit of the model to the data: i.e. how well does the model perform in predicting the outcomes. In his first example is clear that all the models are equally useless: the noise dominates…
Earlier quoted context omitted.
Except that we can't access reality completely. Assuming that there is a "true model" generating data is just that: an assumption with no real basis. Semantics is the essence of communication.
Unless you have data, that is. The data is the basis for assuming that a process has generated data. Either that, or the data has existed for all eternity, and therefore could never have been collected.
I'm a huge fan of R^2 and you should be too. The simple way to think about R^2 is that it is a measure of relative predictive accuracy (and that is exactly how it is calculated). This is both a more accurate (and a more useful definition for most tasks the HN crowd would work on) than saying R^2 is a measure of the distance from the true model. All the figures and findings in the post are completely reasonable given…
I think what John is saying is that people commonly use R^2 as measure of model fit where something like root mean squared error (RMSE) gives a better measure of model fit (by measuring the distance from the true model) depending on the model. Just using R^2 blindly for most tasks you would work on can lead to choosing an incorrect model. I think the main take-away from the post is to better understand the correct me…
I don't mean to be rude, but that is definitely not what he is saying. There are two important things I'd like to clarify:
- It is wrong to call the alternative measure "the RMSE". The alternative that the article was proposing was a made-up measure called E^2 which measures the distance from the true model.
- The author is not suggesting that people use E^2 instead of R^2 for any case. In fact, in almost all cases it is impossible to use E^2, because calculating it requires you to know the true model, and if you know the true model it would be very unlikely that you'd be wasting your time measuring other models.
The author makes it clear that E^2 isn't really to be considered an alternative when he called it the "generally unmeasurable E^2".