Types of Regression Analysis • Hacker News Board

Re: Types of Regression Analysis

#61

Post by michaelbarton » Sat, Apr 14, 2018, 1:20 AM UTC

SubiculumCode wrote: post #27

This article is obviously a jumping off point kind of article. Most people using linear regression have never even heard of things like ridge regression. So I like the article. However, there are at least two types of regression I'd add to the list, and a suggestion.: 1 Multivariate Distance Matrix Regression (MDMR; Anderson, 2001; McArdle & Anderson, 2001). 2. Regression with splines 3. On polynomial regression, add…

There's also hierarchical regression, where you can estimate multilevel models. Also call fixed and random effect models. Assigning a variance coefficient for each parameter can account for heteroscedasticity.

Re: Types of Regression Analysis

#62

Post by syntaxfree » Sat, Apr 14, 2018, 3:52 AM UTC

SubiculumCode wrote: post #45

thanatropism wrote: post #42

Earlier quoted context omitted.

Wrong. Wrong, wrong, wrong, wrong. If predictors are linearly dependent you don't get to do regression at all -- your X'X is singular. But then, the extra regressors add no information at all, and classical statistical packages (SPSS, Stata, etc.) drop them automatically. Even if predictors are highly correlated, the OLS estimator is unbiased . This is the stuff of elementary statistics. You just get lower and lower…

There is no need to be rude or yell. Yes, if your variables are perfectly linearly dependent they get dropped. Did anyone say otherwise? I did not think about this case because most correlated measures causing multicollinearity problems aren't perfectly 'linearly dependent'. Linearly dependency usually only comes up practically if you miscoded some of your independent dummy variables (e.g. adding both 'male[0,1]' and…

There's this other thing called the FWL theorem.

As long as the unexplained term is uncorrelated (in the probabilistic model; linear regression will force this to be the case computationally) with the included variables, your coefficients will remain unchanged. So adding/removing variables shouldn't change results at all -- unless the model is mis-specified and you're including variables that correlate with unobserved factors in unexpected ways.

So for example a regression of children's IQ on the income of their parents provides a plausible mechanism; but if you add the arm length of the kids you will have problems, since arm length is correlated to an omitted variable (kids with longer arms are older and perform better on IQ tests).

That's most of the "in context" story. Nothing to do with multicollinearity.

Re: Types of Regression Analysis

#63

Post by syntaxfree » Sat, Apr 14, 2018, 3:54 AM UTC

SubiculumCode wrote: post #27

This article is obviously a jumping off point kind of article. Most people using linear regression have never even heard of things like ridge regression. So I like the article. However, there are at least two types of regression I'd add to the list, and a suggestion.: 1 Multivariate Distance Matrix Regression (MDMR; Anderson, 2001; McArdle & Anderson, 2001). 2. Regression with splines 3. On polynomial regression, add…

Important for many real use cases such as estimating demand from prices instrumental variable regression.

Re: Types of Regression Analysis

#64

Post by SubiculumCode » Sat, Apr 14, 2018, 4:26 AM UTC

syntaxfree wrote: post #62

SubiculumCode wrote: post #45

Earlier quoted context omitted.

There is no need to be rude or yell. Yes, if your variables are perfectly linearly dependent they get dropped. Did anyone say otherwise? I did not think about this case because most correlated measures causing multicollinearity problems aren't perfectly 'linearly dependent'. Linearly dependency usually only comes up practically if you miscoded some of your independent dummy variables (e.g. adding both 'male[0,1]' and…

There's this other thing called the FWL theorem. As long as the unexplained term is uncorrelated (in the probabilistic model; linear regression will force this to be the case computationally) with the included variables, your coefficients will remain unchanged. So adding/removing variables shouldn't change results at all -- unless the model is mis-specified and you're including variables that correlate with unobserve…

Thanks for the thoughtful comment and reference.

The 'in context' was not so much about multicollinearity but about shared and unique variance.

Re: Types of Regression Analysis

#65

Post by [deleted] » Sat, Apr 14, 2018, 6:59 AM UTC

swebs wrote: post #40

matchagaucho wrote: post #3

Are there any ML APIs or web services that accept a vector and run various regression scenarios to identify optimal fit? I suppose vectors for both training and testing would be required. Would gladly pay $1-$5 per batch for a service to do this.

There is a Python library called TPOT that does this. https://github.com/EpistasisLab/tpot

[deleted]

Re: Types of Regression Analysis

#66

Post by SubiculumCode » Sat, Apr 14, 2018, 10:35 AM UTC

michaelbarton wrote: post #61

SubiculumCode wrote: post #27

This article is obviously a jumping off point kind of article. Most people using linear regression have never even heard of things like ridge regression. So I like the article. However, there are at least two types of regression I'd add to the list, and a suggestion.: 1 Multivariate Distance Matrix Regression (MDMR; Anderson, 2001; McArdle & Anderson, 2001). 2. Regression with splines 3. On polynomial regression, add…

There's also hierarchical regression, where you can estimate multilevel models. Also call fixed and random effect models. Assigning a variance coefficient for each parameter can account for heteroscedasticity.

Mixed models..the hell (+_+) I live in =_=

Re: Types of Regression Analysis

#67

Post by closed » Sat, Apr 14, 2018, 12:07 PM UTC

prestonh wrote: post #49

gmfawcett wrote: post #6

> Assumptions of linear regression: There must be a linear relation between independent and dependent variables. That's not wrong, but it's a strong way to word it. If linear regression were only suitable when the variables were perfectly linearly related, it would get a lot less use. Practically, linear regression can be used when the relationship is linear-ish, at least in the interval of interest. In other words,…

This 'assumption' always bothered me when studying for DS roles because it's something that you're expected to know if asked, but isn't really true/accurate. Another is the non-collinearity assumption between variates, which is violated all the time in ML tasks but an 'assumption' of the model nonetheless.

In general, it's helpful for me to separate assumptions that are characteristics of the generative model (and possibly an inference procedure used with it), and "assumptions" meaning things that could lead to poor out of sample prediction.

Re: Types of Regression Analysis

#68

Post by closed » Sat, Apr 14, 2018, 12:13 PM UTC

MichailP wrote: post #35

cocoablazing wrote: post #34

Earlier quoted context omitted.

You can, but why trash information that is present when you can leverage it with a different approach?

Like PCA? But that way you loose physical meaning of the predictors.

PCA is a special case of factor analysis, so you are representing them as observations of a latent variable (which is often a narrative people use when explaining why two x variables are correlated)

Re: Types of Regression Analysis

#69

Post by nonbel » Sat, Apr 14, 2018, 1:24 PM UTC

siddboots wrote: post #57

nonbel wrote: post #55

Earlier quoted context omitted.

When I tried this to choose xgboost hyperparameters it didn't seem to perform much better than random search while also adding another layer of hyper-hyper-parameters.

Yeah. The hyper parameter story that comes with Gaussian processeses is a big drawback. The choice of kernel has a massive impact. In practice, I've found GPs to be great for getting actual insight into an unknown function, but much less useful as a black-box learner.

I guess at its root the problem may just be how much compute is available to throw at the optimization. Alternatively there could be more efficient algos... I looked into but never fully tested this, it seemed promising: https://news.ycombinator.com/item?id=16241659

Re: Types of Regression Analysis

#70

Post by sfink » Sat, Apr 14, 2018, 3:23 PM UTC

thanatropism wrote: post #16

This is just horrible quality material. What in the heck is this? > It is to be kept in mind that the coefficients which we get in quantile regression for a particular quantile should differ significantly from those we obtain from linear regression. If it is not so then our usage of quantile regression isn't justifiable. This can be done by observing the confidence intervals of regression coefficients of the estimate…

I'm a typical "math is hard; let's go programming" type of person, but the only problem i have with that quoted section is the missing antecedent of "This can be done...". But I worked it out from context.

I thought the article was very good.