Logistic regression from scratch
philippmuens.com
Logistic regression from scratch
1–10 of 65 posts
Re: Logistic regression from scratch
#2My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.
Re: Logistic regression from scratch
#3My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.
This is not quite correct. The log probabilities are
log p(y=1 | x; beta) = beta * x - log Z(x; beta)
where
Z(x) = p(y=0 | x; beta) + p(y=1 | x; beta)
Thus, you can think of it as linear regression, but with an additional term log Z(x; beta) in the log likelihood.
Re: Logistic regression from scratch
#4My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.
From what I recall this is a bit off -- not a bad mental model but the math plays out different.
Linear regression has a closed form solution of X projected onto Y: \hat{\beta} = (X'X)^{-1} X' Y
It is equivalent to the Maximum Likelihood Estimator (MLE) for linear regression. However, for logistic regression, MLE would estimate different from MLE for the log odds output.
Linear regression on {class_inclusion} = XB gives the linear probability model, which has limited utility. The required transform is covered by another commenter.
Re: Logistic regression from scratch
#5My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.
From what I recall this is a bit off -- not a bad mental model but the math plays out different. Linear regression has a closed form solution of X projected onto Y: \hat{\beta} = (X'X)^{-1} X' Y It is equivalent to the Maximum Likelihood Estimator (MLE) for linear regression. However, for logistic regression, MLE would estimate different from MLE for the log odds output. Linear regression on {class_inclusion} = XB gi…
You're right, my model was a bit off. Thanks for pointing that out, I forgot about the fact.
Re: Logistic regression from scratch
#6Logistic regression can learn some quite amazing things. I trained a linear function to play chess:
https://github.com/thomasahle/fastchess
and it manages to predict the next moves of top engine games with 27% accuracy.
A benefit of logistic regression is that the resulting model really fast. Furthermore, it's linear, so you can do incremental updates to your prediction. If you have `n` classes and `b` input features change, you can recompute in `bn` time, rather than doing a full matrix multiplication, which can be a huge time saver.
Re: Logistic regression from scratch
#7Ah, the old "regression from scratch" post that is mandatory for all blogs
Re: Logistic regression from scratch
#8My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.
It's better to think of linear regression and logistic regression as special cases of the Generalized Linear Model (GLM).
In that framework, they are literally the same model with different "settings" - Gaussian vs Bernoulli distribution.
Re: Logistic regression from scratch
#9My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.
> it's linear regression on the log-odds
Almost - logistic regression assumes that the function is linear in the log odds, i.e. log(p/(1-p)) = Xb + e. The problem is that you can't compute the log-odds, because you don't know p.
Re: Logistic regression from scratch
#10My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.
[deleted]