Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Types of Regression Analysis

listendata.com

Post reply on HN
Print view View original

31–40 of 76 posts

Re: Types of Regression Analysis

#31

Post by MichailP »

cocoablazing wrote: post #28
MichailP wrote: post #24

Now this is a topic I desperately need. Can anyone here by any chance explain why would one choose predictors in multilinear regression that are NOT correlated to the target? I am having trouble understanding paper [1] where authors avoid using predictors that are correlated to target. Target is ozone concentration shown by referent instrument and predictors are low cost sensor outputs. [1] https://www.sciencedirect.…

The issue is intra-predictor correlation. In the extreme case that a predictor is duplicated, the correct beta might be {beta a, beta (1-a)} for a in [0, 1], which an algorithm may not estimate in a stable manner. A significant degree of correlation introduces this general problem.

So say you have 3 predictors that have high intra predictor correlation. Can you still pick one of them, and discard the remaning 2? Or you cant pick any one of them?

Re: Types of Regression Analysis

#32

Post by MichailP »

SubiculumCode wrote: post #29
MichailP wrote: post #24

Now this is a topic I desperately need. Can anyone here by any chance explain why would one choose predictors in multilinear regression that are NOT correlated to the target? I am having trouble understanding paper [1] where authors avoid using predictors that are correlated to target. Target is ozone concentration shown by referent instrument and predictors are low cost sensor outputs. [1] https://www.sciencedirect.…

When predictors are correlated with each other you get multicollinearity potentially leading to incorrect statistical inferences.

Thanks for the answer. And what is the correct approach here, if you can only chose/not chose predictor in final set? Discard all multicollinear predictors or pick just one of them?

Re: Types of Regression Analysis

#33

Post by sgt101 »

matchagaucho wrote: post #11
swirepe wrote: post #4

Earlier quoted context omitted.

Why wouldn't you just run weka or something locally?

I just want to provide the data and let a service decide the best algorithm. Weka and various ML tools require you select the algorithm and do the A/B testing on your own. There's an opportunity for an Optimizely of ML.

Model fishing is bad.

You will find a model that looks good on your data. It will not be the model you should use.

Re: Types of Regression Analysis

#34

Post by cocoablazing »

MichailP wrote: post #31
cocoablazing wrote: post #28

Earlier quoted context omitted.

The issue is intra-predictor correlation. In the extreme case that a predictor is duplicated, the correct beta might be {beta a, beta (1-a)} for a in [0, 1], which an algorithm may not estimate in a stable manner. A significant degree of correlation introduces this general problem.

So say you have 3 predictors that have high intra predictor correlation. Can you still pick one of them, and discard the remaning 2? Or you cant pick any one of them?

You can, but why trash information that is present when you can leverage it with a different approach?

Re: Types of Regression Analysis

#35

Post by MichailP »

cocoablazing wrote: post #34
MichailP wrote: post #31

Earlier quoted context omitted.

So say you have 3 predictors that have high intra predictor correlation. Can you still pick one of them, and discard the remaning 2? Or you cant pick any one of them?

You can, but why trash information that is present when you can leverage it with a different approach?

Like PCA? But that way you loose physical meaning of the predictors.

Re: Types of Regression Analysis

#36

Post by beagle3 »

cocoablazing wrote: post #28
MichailP wrote: post #24

Now this is a topic I desperately need. Can anyone here by any chance explain why would one choose predictors in multilinear regression that are NOT correlated to the target? I am having trouble understanding paper [1] where authors avoid using predictors that are correlated to target. Target is ozone concentration shown by referent instrument and predictors are low cost sensor outputs. [1] https://www.sciencedirect.…

The issue is intra-predictor correlation. In the extreme case that a predictor is duplicated, the correct beta might be {beta a, beta (1-a)} for a in [0, 1], which an algorithm may not estimate in a stable manner. A significant degree of correlation introduces this general problem.

... or worse; it is still true for any a. You could easily get {1,000,001, -1,000,000}, which for perfectly clean, precise, representable data is equivalent, but which magnifies any noise/error in one of the predictors by a million. or a billion.

Re: Types of Regression Analysis

#37

Post by SubiculumCode »

MichailP wrote: post #32
SubiculumCode wrote: post #29

Earlier quoted context omitted.

When predictors are correlated with each other you get multicollinearity potentially leading to incorrect statistical inferences.

Thanks for the answer. And what is the correct approach here, if you can only chose/not chose predictor in final set? Discard all multicollinear predictors or pick just one of them?

Keeping just to linear regression. If those variables are measuring the same construct, pick the best one or use a method to combine their scores. If they measure different constructs but are very correlated, then you'd need to drop one..depending on the variance inflation factor...which you can test for.

As the article mentions however, there are regression methods meant for these situations (e.g. ridge regression).

Re: Types of Regression Analysis

#38

Post by beagle3 »

MichailP wrote: post #31
cocoablazing wrote: post #28

Earlier quoted context omitted.

The issue is intra-predictor correlation. In the extreme case that a predictor is duplicated, the correct beta might be {beta a, beta (1-a)} for a in [0, 1], which an algorithm may not estimate in a stable manner. A significant degree of correlation introduces this general problem.

So say you have 3 predictors that have high intra predictor correlation. Can you still pick one of them, and discard the remaning 2? Or you cant pick any one of them?

Using ridge regression (mentioned in TFA) would prefer a (1/3,1/3,1/3) average of those predictors (or a better combination, depending on their respective noises).

Using lasso (also mentioned in TFA) would prefer to pick the best of the three and drop the others.

Using elastic net would be a combination of both.

Note, though, that any method other than simple regression has tuning parameters -- depending on those, you could still end with result equivalent to plain least squares.

Re: Types of Regression Analysis

#39

Post by SubiculumCode »

MichailP wrote: post #32
SubiculumCode wrote: post #29

Earlier quoted context omitted.

When predictors are correlated with each other you get multicollinearity potentially leading to incorrect statistical inferences.

Thanks for the answer. And what is the correct approach here, if you can only chose/not chose predictor in final set? Discard all multicollinear predictors or pick just one of them?

One thing that should be mentioned though is in the case of polynomials e.g. y ~ x + x^2, there will be a lot of multicollinearity between these terms, but that multicollinearity is OK...just be sure to center your variables.

Re: Types of Regression Analysis

#40

Post by swebs »

matchagaucho wrote: post #3

Are there any ML APIs or web services that accept a vector and run various regression scenarios to identify optimal fit? I suppose vectors for both training and testing would be required. Would gladly pay $1-$5 per batch for a service to do this.

There is a Python library called TPOT that does this.

https://github.com/EpistasisLab/tpot

Post reply on HN