Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logistic regression from scratch

philippmuens.com

Post reply on HN
Print view View original

11–20 of 65 posts

Re: Logistic regression from scratch

#11

Post by j7ake »

IdiocyInAction wrote: post #2

My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.

Putting logistic and linear regression into the generalised linear model framework is the right way to think of it and compare them.

From this point of view linear regression would be using GLM with identity link function, logistic regression uses the logit function as the link function.

Re: Logistic regression from scratch

#12

Post by oli5679 »

Weight of evidence binning can be helpful feature engineering strategy for logistic regression.

Often this is a good 'first cut' model for a binary classifier on tabular data. If feature interactions don't have a major impact on your target then this can actually be a tough benchmark to beat.

https://github.com/oli5679/WeightOfEvidenceDemo

https://www.listendata.com/2015/03/weight-of-evidence-woe-an...

Re: Logistic regression from scratch

#14

Post by mhh__ »

thomasahle wrote: post #6

Logistic regression can learn some quite amazing things. I trained a linear function to play chess: https://github.com/thomasahle/fastchess and it manages to predict the next moves of top engine games with 27% accuracy. A benefit of logistic regression is that the resulting model really fast. Furthermore, it's linear, so you can do incremental updates to your prediction. If you have `n` classes and `b` input features…

Isn't 27% worse than flipping a coin?

Re: Logistic regression from scratch

#15

Post by gpderetta »

mhh__ wrote: post #14
thomasahle wrote: post #6

Logistic regression can learn some quite amazing things. I trained a linear function to play chess: https://github.com/thomasahle/fastchess and it manages to predict the next moves of top engine games with 27% accuracy. A benefit of logistic regression is that the resulting model really fast. Furthermore, it's linear, so you can do incremental updates to your prediction. If you have `n` classes and `b` input features…

Isn't 27% worse than flipping a coin?

Only if you have only two legal moves.

Re: Logistic regression from scratch

#16

Post by enchiridion »

mhh__ wrote: post #14
thomasahle wrote: post #6

Logistic regression can learn some quite amazing things. I trained a linear function to play chess: https://github.com/thomasahle/fastchess and it manages to predict the next moves of top engine games with 27% accuracy. A benefit of logistic regression is that the resulting model really fast. Furthermore, it's linear, so you can do incremental updates to your prediction. If you have `n` classes and `b` input features…

Isn't 27% worse than flipping a coin?

No, uniform random would be bounded by 1/16. However you cannot move ever piece in every configuration, so it's greater than that. Actually would be an interesting problem for figure out...

Re: Logistic regression from scratch

#17

Post by thomasahle »

mhh__ wrote: post #14
thomasahle wrote: post #6

Logistic regression can learn some quite amazing things. I trained a linear function to play chess: https://github.com/thomasahle/fastchess and it manages to predict the next moves of top engine games with 27% accuracy. A benefit of logistic regression is that the resulting model really fast. Furthermore, it's linear, so you can do incremental updates to your prediction. If you have `n` classes and `b` input features…

Isn't 27% worse than flipping a coin?

A typical chess position has 20-40 legal moves. The complete space of moves for the model to predict from has about 1800 moves.

For comparison Leela Zero gets around 60% accuracy on predicting its own next move.

With this sort of accuracy you can reduce the search part of the algorithm to an effective branching factor of 2-4 rather than 40, nearly for free, which is a pretty big win.

Re: Logistic regression from scratch

#18

Post by colinmhayes »

How do we differentiate between econometrics and machine learning? Logistic regression seems like it fits into econometrics better than machine learning to me. There's no regularization. I guess there's gradient descent which can be seen as more machine learning. In the end it's semantics of course, still an interesting distinction.

Re: Logistic regression from scratch

#19

Post by random314 »

IdiocyInAction wrote: post #2

My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.

Linear regression uses MSE loss. Logistic regression uses log-loss. Both loss functions behave differently.

Its not just the underlying model, but the loss function is also different.

Re: Logistic regression from scratch

#20

Post by random314 »

colinmhayes wrote: post #18

How do we differentiate between econometrics and machine learning? Logistic regression seems like it fits into econometrics better than machine learning to me. There's no regularization. I guess there's gradient descent which can be seen as more machine learning. In the end it's semantics of course, still an interesting distinction.

Logistic regression can use both L1 and L2 regularization
Post reply on HN