Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Types of Regression Analysis

listendata.com

Post reply on HN
Print view View original

41–50 of 76 posts

Re: Types of Regression Analysis

#41

Post by bunderbunder »

meanmrmustard92 wrote: post #22
bunderbunder wrote: post #17

Earlier quoted context omitted.

It's not wrong, it's just a way of looking at things that speaks to the underlying math rather than the full extent of what you can do with it if you extend it with things like kernel methods. When you use linear regression to fit a model like y ~ ax + b(x^2) what you're technically doing is fitting a linear function with two parameters on two variables. One variable happens to always be equal to the square of the ot…

Fair enough. Mechanically, all you're ever doing when estimating a parameter vector using OLS is projecting Y onto the span of X, and that requires linearity in the sense that Y = XB. But far too often I've met people who've come away thinking OLS is useless because they mistake the linearity in parameters with 'y must be a linear function of x', which is they think is too simplistic, and so they go do more complicat…

To me, that's a stellar example of why you probably shouldn't have people who don't even have a basic undergraduate "intro to stats" understanding of the subject doing your statistics work.

I get that it's a potential cause of confusion for someone who has no training in stats. But it's also jargon that describes a useful concept, and that is literally transparent if you do have enough understanding of the math to know what "linear" and "parameters" mean in this context.

Re: Types of Regression Analysis

#42

Post by thanatropism »

SubiculumCode wrote: post #29
MichailP wrote: post #24

Now this is a topic I desperately need. Can anyone here by any chance explain why would one choose predictors in multilinear regression that are NOT correlated to the target? I am having trouble understanding paper [1] where authors avoid using predictors that are correlated to target. Target is ozone concentration shown by referent instrument and predictors are low cost sensor outputs. [1] https://www.sciencedirect.…

When predictors are correlated with each other you get multicollinearity potentially leading to incorrect statistical inferences.

Wrong.

Wrong, wrong, wrong, wrong.

If predictors are linearly dependent you don't get to do regression at all -- your X'X is singular. But then, the extra regressors add no information at all, and classical statistical packages (SPSS, Stata, etc.) drop them automatically.

Even if predictors are highly correlated, the OLS estimator is unbiased. This is the stuff of elementary statistics. You just get lower and lower p-values/wider and wider CIs, specially if your samples are econometrics-sized.

---

You people need to watch some Khan Academy or whatever the cool kids are doing now to learn maths.

Re: Types of Regression Analysis

#43

Post by mnky9800n »

Logistic regression is doing classification not regression. That is, it's assigning/predicting categories of data points instead of predicting some continuous value on some interval. Maybe this is splitting hairs but the way you evaluate a classification model is totally different than a regression one.

Re: Types of Regression Analysis

#44

Post by myopicgoat »

mnky9800n wrote: post #43

Logistic regression is doing classification not regression. That is, it's assigning/predicting categories of data points instead of predicting some continuous value on some interval. Maybe this is splitting hairs but the way you evaluate a classification model is totally different than a regression one.

This is a question of perspective, you can in fact just take it as a regression over the continuous 0-1 interval with loss |y-o(Xw)| where o is the sigmoid and could report that loss (in fact a package such as Sklearn will usually return the parameter values that minimise that loss, possibly with a penalty). If you want to use it as a classifier then you threshold the predictions.

Re: Types of Regression Analysis

#45

Post by SubiculumCode »

thanatropism wrote: post #42
SubiculumCode wrote: post #29

Earlier quoted context omitted.

When predictors are correlated with each other you get multicollinearity potentially leading to incorrect statistical inferences.

Wrong. Wrong, wrong, wrong, wrong. If predictors are linearly dependent you don't get to do regression at all -- your X'X is singular. But then, the extra regressors add no information at all, and classical statistical packages (SPSS, Stata, etc.) drop them automatically. Even if predictors are highly correlated, the OLS estimator is unbiased . This is the stuff of elementary statistics. You just get lower and lower…

There is no need to be rude or yell.

Yes, if your variables are perfectly linearly dependent they get dropped. Did anyone say otherwise? I did not think about this case because most correlated measures causing multicollinearity problems aren't perfectly 'linearly dependent'. Linearly dependency usually only comes up practically if you miscoded some of your independent dummy variables (e.g. adding both 'male[0,1]' and 'not male[0,1]' as two categorical predictors). So I am not really sure of your point.

As to your second point, it might be unbiased but the statistical inference (i.e. p-value) would be incorrect with multi-collinearity..thus again, I am not sure of your point when you are only repeating what I said.

Moreover, it may not be particularly meaningful to the researcher even if the parameter estimate is unbiased. One frequently finds with multicollinearity that the signs of effects will switch (- to +, or + to -) as you add highly correlated predictors into a model, in oft-theoretically questionable ways, but does serve to remind one that the parameter estimates are only meaningful in the context of the other predictors in the model.

Re: Types of Regression Analysis

#46

Post by smu3l »

mnky9800n wrote: post #43

Logistic regression is doing classification not regression. That is, it's assigning/predicting categories of data points instead of predicting some continuous value on some interval. Maybe this is splitting hairs but the way you evaluate a classification model is totally different than a regression one.

This is not correct. Logistic regression can be used for classification, true, but it can also be viewed as a way of estimating the conditional mean of an outcome variable that has a Bernoulli, or binomial distribution, depending on the formulation.

There are many ways to evaluate all of these methods, and for classification you may favor something else, but it's completely reasonable to use the (cross validated, or not) empirical risk for both logistic and linear regression. That would be a negative log likelihood in both cases, from the Bernoulli/binomial distribution for logistic regression or the normal distribution for linear regression.

Re: Types of Regression Analysis

#47

Post by smu3l »

nerdponx wrote: post #9
gmfawcett wrote: post #6

> Assumptions of linear regression: There must be a linear relation between independent and dependent variables. That's not wrong, but it's a strong way to word it. If linear regression were only suitable when the variables were perfectly linearly related, it would get a lot less use. Practically, linear regression can be used when the relationship is linear-ish, at least in the interval of interest. In other words,…

That the linear model is "correct" is only assumption if you're trying to draw probabilistic inferences. There's nothing stopping you from using it as a "best fit line", even when you have no reason to believe those assumptions. But then it's just a best-fit line. It tells you the direction and magnitude of linear trend, nothing more. That's never wrong in any sense, it's just that sometimes it's not very useful.

From the semiparametric perspective, you can still make correct inferences about the estimated parameters even if the model is not correctly specified, as long as you use the so-called robust estimator of the variance.

Re: Types of Regression Analysis

#48

Post by your-nanny »

thanatropism wrote: post #42
SubiculumCode wrote: post #29

Earlier quoted context omitted.

When predictors are correlated with each other you get multicollinearity potentially leading to incorrect statistical inferences.

Wrong. Wrong, wrong, wrong, wrong. If predictors are linearly dependent you don't get to do regression at all -- your X'X is singular. But then, the extra regressors add no information at all, and classical statistical packages (SPSS, Stata, etc.) drop them automatically. Even if predictors are highly correlated, the OLS estimator is unbiased . This is the stuff of elementary statistics. You just get lower and lower…

Watta jerk.

Re: Types of Regression Analysis

#49

Post by prestonh »

gmfawcett wrote: post #6

> Assumptions of linear regression: There must be a linear relation between independent and dependent variables. That's not wrong, but it's a strong way to word it. If linear regression were only suitable when the variables were perfectly linearly related, it would get a lot less use. Practically, linear regression can be used when the relationship is linear-ish, at least in the interval of interest. In other words,…

This 'assumption' always bothered me when studying for DS roles because it's something that you're expected to know if asked, but isn't really true/accurate. Another is the non-collinearity assumption between variates, which is violated all the time in ML tasks but an 'assumption' of the model nonetheless.

Re: Types of Regression Analysis

#50

Post by nonbel »

smu3l wrote: post #47
nerdponx wrote: post #9

Earlier quoted context omitted.

That the linear model is "correct" is only assumption if you're trying to draw probabilistic inferences. There's nothing stopping you from using it as a "best fit line", even when you have no reason to believe those assumptions. But then it's just a best-fit line. It tells you the direction and magnitude of linear trend, nothing more. That's never wrong in any sense, it's just that sometimes it's not very useful.

From the semiparametric perspective, you can still make correct inferences about the estimated parameters even if the model is not correctly specified, as long as you use the so-called robust estimator of the variance.

This is impossible. If the model is incorrectly specified (does not include all and only the relevant parameters and interactions), it doesn't matter much what games are played with the math. Changing the model will change the estimates...

Edit: For example, see here where making arbitrary choices of how to code categorical variables will change the estimates: https://news.ycombinator.com/item?id=16719754

If you change the model the meaning of all the coefficients changes.

Post reply on HN