Live data from Hacker News

Logistic regression from scratch

philippmuens.com

21–30 of 65 posts

Re: Logistic regression from scratch

#21
post #14

Earlier quoted context omitted.

Isn't 27% worse than flipping a coin?

A typical chess position has 20-40 legal moves. The complete space of moves for the model to predict from has about 1800 moves. For comparison Leela Zero gets around 60% accuracy on predicting its own next move. With this sort of accuracy you can reduce the search part of the algorithm to an effective branching factor of 2-4 rather than 40, nearly for free, which is a pretty big win.

I don’t understand the comment about Leela. Why isn’t own move prediction deterministic?

Re: Logistic regression from scratch

#22

How do we differentiate between econometrics and machine learning? Logistic regression seems like it fits into econometrics better than machine learning to me. There's no regularization. I guess there's gradient descent which can be seen as more machine learning. In the end it's semantics of course, still an interesting distinction.

Logistic regression can use both L1 and L2 regularization

And ols can too. That doesn't make it machine learning. This implementation doesn't involve any regularization.

Re: Logistic regression from scratch

#23

How do we differentiate between econometrics and machine learning? Logistic regression seems like it fits into econometrics better than machine learning to me. There's no regularization. I guess there's gradient descent which can be seen as more machine learning. In the end it's semantics of course, still an interesting distinction.

From my experience, econometricians and ML practitioners mostly pretend like the other group doesn't exist.

Re: Logistic regression from scratch

#24

My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.

There is a different mathematical justification as well in terms of Bayesian reasoning.

The claim is that “evidence” in Bayesian reasoning naturally acts upon the log-odds, mapping prior log-odds to posterior log-odds additively. To see this, calculate from the definition,

    Odds(X | E) := Pr(X | E) / Pr(¬X | E)
                 = Pr(X ∧ E) / Pr(¬X ∧ E)
                 = (Pr(E | X) × Pr(X)) / (Pr(E | ¬X) × Pr(¬X))
                 = LR(E, X) × Odds(X),
Where LR is the usual likelihood ratio.

So when we take the logarithm of both sides, we find that new evidence adds some quantity—the log of the likelihood ratio of the evidence—to our log of prior probability, in this phrasing of Bayes’ theorem.

I sometimes tell people this in a slightly strange language, I say that if we ran into aliens we might find out that they don't believe the things are absolutely true or false, but instead measure their truth or falsity in decibels.

So another perspective on what logistic regression is trying to do, is that it is trying to assume linear log-likelihood-ratio dependence based on the strength of some independent pieces of evidence. You can weakly justify this in all cases, using calculus and assuming everything has a small impact. You can further justify it strongly for any signal where twice as large of a measured regression variable ultimately implies twice as many independent events at a much lower level happening and independently providing their evidences for the regression outcome. So like, I come from a physics background, I am thinking in this case of photon counts in a photomultiplier tube or so: I know that at a lower level, each photon is contributing equally some small little bit of evidence for something, so when I count all the time up together, this is the appropriate framework to use.

Re: Logistic regression from scratch

#25

How do we differentiate between econometrics and machine learning? Logistic regression seems like it fits into econometrics better than machine learning to me. There's no regularization. I guess there's gradient descent which can be seen as more machine learning. In the end it's semantics of course, still an interesting distinction.

Well, what do you define as machine learning?

Logistic regression is clearly a classifier. And you need data to train it. So it's a supervised learning algorithm.

Re: Logistic regression from scratch

#26

How do we differentiate between econometrics and machine learning? Logistic regression seems like it fits into econometrics better than machine learning to me. There's no regularization. I guess there's gradient descent which can be seen as more machine learning. In the end it's semantics of course, still an interesting distinction.

Well, what do you define as machine learning? Logistic regression is clearly a classifier. And you need data to train it. So it's a supervised learning algorithm.

I'm trying to have a conversation so I can figure it out. Pretty confident that being a classifier does not make it machine learning, econometrics has classifiers too. Econometric models also need data to train them, so I'm not sure your second point is helpful either. Unless you're claiming the difference is nothing but whether the model is used by an economist.

Re: Logistic regression from scratch

#27
Very interesting seeing people in the comments debate what is a very basic thing taught in any stats/econometrics class.

The idea behind binary regression ( Y is 0 or 1) is that you use a latent variable Y* = beta X + epsilon.

X is the matrix of indenpent variables, beta is the vector of coefficients and epsilon is an error term that sums the rest of what X can't explain.

Y thus becomes 1 if Y* >0 and 0 otherwise.

Seeing how Y is binary, we can model it using a Bernoulli distribution with a success probability P(Y=1) = P(Y* >0) = 1 - P(Y* Technically you can use any function that maps R to [0, 1] as a CDF. If its density is symmetrical then you can directly write the above probability as CDF(beta X). The two usual choices are either the normal CDF which gives the Probit model or the Logistic function (Sigmoid) which gives the Logit model. With the CDF known you can calculate the likelihood and use it to estimate the coefficients.

People prefer the Logit model because the coefficients of the model are interpretable in terms of log-odds and all and the fucntion has some nice numerical properties.

That's all there is to it really.

Re: Logistic regression from scratch

#28

Earlier quoted context omitted.

A typical chess position has 20-40 legal moves. The complete space of moves for the model to predict from has about 1800 moves. For comparison Leela Zero gets around 60% accuracy on predicting its own next move. With this sort of accuracy you can reduce the search part of the algorithm to an effective branching factor of 2-4 rather than 40, nearly for free, which is a pretty big win.

I don’t understand the comment about Leela. Why isn’t own move prediction deterministic?

Because Leela (like fastchess mentioned above) has two parts: A neural network predicting good moves, and a tree search exploring the moves suggested and evaluating the resulting positions (with a second net).

If the prediction (policy) net had a 100% accuracy, you wouldn't need the tree search part at all.

Re: Logistic regression from scratch

#29
post #14

Earlier quoted context omitted.

Isn't 27% worse than flipping a coin?

No, uniform random would be bounded by 1/16. However you cannot move ever piece in every configuration, so it's greater than that. Actually would be an interesting problem for figure out...

There are only 16 pieces, but in most board positions, many pieces can make more than one legal move.

Re: Logistic regression from scratch

#30
post #8

My understanding of Logistic Regression is that it's linear regression on the log-odds, which are then converted to probabilities with the sigmoid/softmax function. This formulation allows one to do direct linear regression on the probabilities, without the unpleasant side effects of just using a linear model as-is. A mathematical justification for doing this is given by the generalized linear model formulation.

It's better to think of linear regression and logistic regression as special cases of the Generalized Linear Model (GLM). In that framework, they are literally the same model with different "settings" - Gaussian vs Bernoulli distribution.

I have to disagree with you. While assuming Gaussian disturbance terms results in a linear regression, the linear regression framework is more general. It makes no assumptions about the distribution of the disturbance terms. Instead, it merely restricts the variance to be constant over all values of the response variable.
Post reply on HN