[deleted]
Types of Regression Analysis
71–76 of 76 posts
Re: Types of Regression Analysis
#72Earlier quoted context omitted.
When I tried this to choose xgboost hyperparameters it didn't seem to perform much better than random search while also adding another layer of hyper-hyper-parameters.
Yeah. The hyper parameter story that comes with Gaussian processeses is a big drawback. The choice of kernel has a massive impact. In practice, I've found GPs to be great for getting actual insight into an unknown function, but much less useful as a black-box learner.
What kernels would you recommend trying initially? I’m still unclear if the Gaussian processes require normal distribution (e.g. would they work on log-log / binomial based functions).
I’ve wanted to apply the approach you mention a few times, but documentation seems to go from “Wiki” level to novel research articles. Are there any good introductory books / resources that aren’t beginner level? That scikit library looks handy!
Re: Types of Regression Analysis
#73Why did the article cover a basic term like "outlier" under "Terminologies related to regression" but omitted information about how to evaluate a regression model? I liked that there was some information at the bottom about "How to choose a regression model" that mentioned "you can select the final model based on Adjusted r-square, RMSE, AIC and BIC" but providing a little more context would make this post even better. Perhaps a link to a future blog post on the topic?
Re: Types of Regression Analysis
#74Earlier quoted context omitted.
From the semiparametric perspective, you can still make correct inferences about the estimated parameters even if the model is not correctly specified, as long as you use the so-called robust estimator of the variance.
This is impossible. If the model is incorrectly specified (does not include all and only the relevant parameters and interactions), it doesn't matter much what games are played with the math. Changing the model will change the estimates... Edit: For example, see here where making arbitrary choices of how to code categorical variables will change the estimates: https://news.ycombinator.com/item?id=16719754 If you chan…
Oops I did not see your response until now.
I agree, changing the model changes the estimates, because the parameters you are estimating change.
However, given one misspecified model, the parameters of that model are still well defined, though they may not have the interpretation they would if the model was correctly specified. As OP called it, this is the "best fit line", and is a projection of the truth onto your model. E.g. for a simple linear regression of Y on X, where the true conditional mean of Y given X is not linear, there is still some "true" best line. This line depends also on the distribution of X, though it would not if the model was correct. Estimates from linear regression will converge to the parameters of this line, though using the usual standard errors will be wrong.
There's a very general theorem or corollary that covers this in Asymptotic Statistics by van der Vaart. I think in the chapter about M estimators, right around where MLEs are covered, but I don't have it in front of me.
Re: Types of Regression Analysis
#75Earlier quoted context omitted.
This is impossible. If the model is incorrectly specified (does not include all and only the relevant parameters and interactions), it doesn't matter much what games are played with the math. Changing the model will change the estimates... Edit: For example, see here where making arbitrary choices of how to code categorical variables will change the estimates: https://news.ycombinator.com/item?id=16719754 If you chan…
Oops I did not see your response until now. I agree, changing the model changes the estimates, because the parameters you are estimating change. However, given one misspecified model, the parameters of that model are still well defined, though they may not have the interpretation they would if the model was correctly specified. As OP called it, this is the "best fit line", and is a projection of the truth onto your m…
There are multiple inference levels here.
First, there is the statistical level, at which we are drawing some conclusion about the model parameter. This may work even for a misspecified model.
Then there is the level at which you want to draw some conclusion about reality, call it the "scientific level". If the model is misspecified, the parameters/coefficients may or may not correspond to the thing of interest. Perhaps the model is a close enough approximation for those values to be meaningful, perhaps not...
I think it is the second ("scientific level") of inference that most people are concerned about. The rigor of the proofs/theorems that may work at the statistical level does not extend to the scientific level.
Afaict, the majority of erroneous inference occurs at the scientific level and statistical error/uncertainty is a sort of minimum error/uncertainty.
Re: Types of Regression Analysis
#76Earlier quoted context omitted.
Yeah. The hyper parameter story that comes with Gaussian processeses is a big drawback. The choice of kernel has a massive impact. In practice, I've found GPs to be great for getting actual insight into an unknown function, but much less useful as a black-box learner.
What kernels would you recommend trying initially? I’m still unclear if the Gaussian processes require normal distribution (e.g. would they work on log-log / binomial based functions). I’ve wanted to apply the approach you mention a few times, but documentation seems to go from “Wiki” level to novel research articles. Are there any good introductory books / resources that aren’t beginner level? That scikit library lo…
Gaussianprocess.org