Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logistic regression from scratch

philippmuens.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Logistic regression from scratch

#41

Post by curiousgal »

reactspa wrote: post #40
curiousgal wrote: post #27

Very interesting seeing people in the comments debate what is a very basic thing taught in any stats/econometrics class. The idea behind binary regression ( Y is 0 or 1) is that you use a latent variable Y* = beta X + epsilon. X is the matrix of indenpent variables, beta is the vector of coefficients and epsilon is an error term that sums the rest of what X can't explain. Y thus becomes 1 if Y* >0 and 0 otherwise. Se…

I've always found it a little simplistic that the default cut-off, in most statistical software, for whether something should be 0 or 1, is 0.5. (i.e. > 0.5 equals 1, and This seems to be a "rarely-questioned assumption". Is there a reason why this is considered reasonable? And is there a name for the cut-off (i.e., if I were to want to change the cut-off, what keyword should I search for inside the software's manual…

From a stats perspective the cutoff is included in the coefficients. If you use a design matrix (add a column of 1s to your variables) you get in a non matrix notation (beta_01 + beta_1 X_1 +...) So the threshold can be considered beta_0.

In the software, you can get classification models to output class probabilities instead of class labels. You can then use whatever threshold you like for to transform those probabilities to labels.

You may see it refered to as "discrimination threshold". Varying that threshold is how ROC curves are constructed.

Re: Logistic regression from scratch

#42

Post by [deleted] »

reactspa wrote: post #40
curiousgal wrote: post #27

Very interesting seeing people in the comments debate what is a very basic thing taught in any stats/econometrics class. The idea behind binary regression ( Y is 0 or 1) is that you use a latent variable Y* = beta X + epsilon. X is the matrix of indenpent variables, beta is the vector of coefficients and epsilon is an error term that sums the rest of what X can't explain. Y thus becomes 1 if Y* >0 and 0 otherwise. Se…

I've always found it a little simplistic that the default cut-off, in most statistical software, for whether something should be 0 or 1, is 0.5. (i.e. > 0.5 equals 1, and This seems to be a "rarely-questioned assumption". Is there a reason why this is considered reasonable? And is there a name for the cut-off (i.e., if I were to want to change the cut-off, what keyword should I search for inside the software's manual…

[deleted]

Re: Logistic regression from scratch

#43

Post by curiousgal »

platz wrote: post #39
curiousgal wrote: post #27

Very interesting seeing people in the comments debate what is a very basic thing taught in any stats/econometrics class. The idea behind binary regression ( Y is 0 or 1) is that you use a latent variable Y* = beta X + epsilon. X is the matrix of indenpent variables, beta is the vector of coefficients and epsilon is an error term that sums the rest of what X can't explain. Y thus becomes 1 if Y* >0 and 0 otherwise. Se…

fyi, generally when explaining things to non-practitioners, it's a detractor to add qualifiers like - this is a very basic thing - that's all there is to it because although it is a basic thing to you, it's not a basic thing to someone who hasn't spent the same time studying all the concepts beforehand. This is generally why you have to take a class to grok stats rather than just read some reference material and defi…

I totally agree but I was explaining that to practionners, or at least people who consider themselves data scientists. I agree that it is by no means a beginner friendly explanation.

Re: Logistic regression from scratch

#44

Post by ImaCake »

platz wrote: post #39
curiousgal wrote: post #27

Very interesting seeing people in the comments debate what is a very basic thing taught in any stats/econometrics class. The idea behind binary regression ( Y is 0 or 1) is that you use a latent variable Y* = beta X + epsilon. X is the matrix of indenpent variables, beta is the vector of coefficients and epsilon is an error term that sums the rest of what X can't explain. Y thus becomes 1 if Y* >0 and 0 otherwise. Se…

fyi, generally when explaining things to non-practitioners, it's a detractor to add qualifiers like - this is a very basic thing - that's all there is to it because although it is a basic thing to you, it's not a basic thing to someone who hasn't spent the same time studying all the concepts beforehand. This is generally why you have to take a class to grok stats rather than just read some reference material and defi…

The author of the parent explains their view well I think (not directed at statistics students). But I would like to make the more general complaint that this attitude seems pervasive in maths. People often fail to include enough detail in their derivations of proofs or their explanations of things and the students are left confused and unsure because they don't see how they got from step 3 to step 4.

Re: Logistic regression from scratch

#45

Post by anthony_doan »

reactspa wrote: post #40
curiousgal wrote: post #27

Very interesting seeing people in the comments debate what is a very basic thing taught in any stats/econometrics class. The idea behind binary regression ( Y is 0 or 1) is that you use a latent variable Y* = beta X + epsilon. X is the matrix of indenpent variables, beta is the vector of coefficients and epsilon is an error term that sums the rest of what X can't explain. Y thus becomes 1 if Y* >0 and 0 otherwise. Se…

I've always found it a little simplistic that the default cut-off, in most statistical software, for whether something should be 0 or 1, is 0.5. (i.e. > 0.5 equals 1, and This seems to be a "rarely-questioned assumption". Is there a reason why this is considered reasonable? And is there a name for the cut-off (i.e., if I were to want to change the cut-off, what keyword should I search for inside the software's manual…

statistical software never does this.

Almost all statistical models give you probabilities and it is up to the domain to determine the cutoff. You can clearly see this in logistic regression it's giving you a probability not a 1 or 0.

When a clinician gives me a dataset I build the model and leave it to them to do the cut off. It is not my domain or expertise to tell them where the cut off is. Plus it gives away the responsibility. I'm not responsible for it unless they don't know and are asking me for a reasonable cut off.

An example is people opting for artificial in semenation because of problem. I believe the cut off is above 60% because of the trade off.

You can read more about it in Regression Modeling Strategies by Dr. Frank Harrell.

Re: Logistic regression from scratch

#46

Post by kelvin0 »

curiousgal wrote: post #43
platz wrote: post #39

Earlier quoted context omitted.

fyi, generally when explaining things to non-practitioners, it's a detractor to add qualifiers like - this is a very basic thing - that's all there is to it because although it is a basic thing to you, it's not a basic thing to someone who hasn't spent the same time studying all the concepts beforehand. This is generally why you have to take a class to grok stats rather than just read some reference material and defi…

I totally agree but I was explaining that to practionners, or at least people who consider themselves data scientists. I agree that it is by no means a beginner friendly explanation.

Is that what some people call the 'mansplain' method?

Re: Logistic regression from scratch

#47

Post by iamstupidsimple »

colinmhayes wrote: post #18

How do we differentiate between econometrics and machine learning? Logistic regression seems like it fits into econometrics better than machine learning to me. There's no regularization. I guess there's gradient descent which can be seen as more machine learning. In the end it's semantics of course, still an interesting distinction.

I think you'll enjoy this delightful thread: https://stackoverflow.com/questions/4205105/whats-the-differ...

Re: Logistic regression from scratch

#48

Post by zwaps »

nerdponx wrote: post #8
IdiocyInAction wrote: post #2

My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.

It's better to think of linear regression and logistic regression as special cases of the Generalized Linear Model (GLM). In that framework, they are literally the same model with different "settings" - Gaussian vs Bernoulli distribution.

Well, then I am gonna say it's even better to think of linear and logistic regression as special case of M Estimators or GMM.

Joke aside, the truth is that logistic regression can be understood based on several assumptions.

Above we have the latent variable explanation. Then there is a Bayesian version. There's even a "random utlity" formulation, where one models explicitly choices of an agent with a probabilistic error. That one is good to explain hierarchical logit models and many of the "issues" with logit such as IAA.

GLM on the other hand I don't feel like it adds much except parameterizing the procedure, which ain't even a good thing. Nowadays we appreciate the semi parametric nature of regression a lot, which is why GLM has declined in use.

Re: Logistic regression from scratch

#49

Post by zwaps »

nerdponx wrote: post #36
markkvdb wrote: post #30

Earlier quoted context omitted.

I have to disagree with you. While assuming Gaussian disturbance terms results in a linear regression, the linear regression framework is more general. It makes no assumptions about the distribution of the disturbance terms. Instead, it merely restricts the variance to be constant over all values of the response variable.

Both things can be true. Linear regression is extra-special because it's a special case of several different frameworks and model classes. I should have written that it's better (in my opinion) to think of logistic regression in the context of GLMs, at least while you're learning. Edit: yes logistic regression is a special case of regression with a different loss function. But it's not nearly "as special" as linear r…

As above, I would strongly agree with you. Both linear and logistic regression can be special cases of frameworks that are more general and far less parametric than GLM. But they also have very intuitive or hands-on explanations, especially logistic regression, which GLM doesn't have.

Re: Logistic regression from scratch

#50

Post by cuchoi »

curiousgal wrote: post #41
reactspa wrote: post #40

Earlier quoted context omitted.

I've always found it a little simplistic that the default cut-off, in most statistical software, for whether something should be 0 or 1, is 0.5. (i.e. > 0.5 equals 1, and This seems to be a "rarely-questioned assumption". Is there a reason why this is considered reasonable? And is there a name for the cut-off (i.e., if I were to want to change the cut-off, what keyword should I search for inside the software's manual…

From a stats perspective the cutoff is included in the coefficients. If you use a design matrix (add a column of 1s to your variables) you get in a non matrix notation (beta_0 1 + beta_1 X_1 +...) So the threshold can be considered beta_0. In the software, you can get classification models to output class probabilities instead of class labels. You can then use whatever threshold you like for to transform those probab…

The threshold would be beta_0 on every case or only when you have subtracted the mean from your data?
Post reply on HN