Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logistic regression from scratch

philippmuens.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Logistic regression from scratch

#51

Post by mr_toad »

colinmhayes wrote: post #37
eVoLInTHRo wrote: post #35

Earlier quoted context omitted.

Econometrics is the application of statistical techniques on economics-related problems, typically to understand relationships between economic phenomena (e.g. income) and things that might be associated with it (e.g. education). Machine learning is typically defined as a way to enable computers to learn from data to accomplish tasks, without explicitly telling them how. Both fields can use logistic regression, regul…

So you're going with the only difference being who's building the model. Interesting take, can't say I disagree much. Although I would say that regularization in econometric models is a bit rare because it distorts the coefficients which as you pointed out is the primary goal of econometrics.

Econometric models tend to be hand-fit and focus more on explanation/hypothesis testing than prediction, so automated variable selection is less common (and sometimes frowned upon).

Re: Logistic regression from scratch

#52

Post by chandra381 »

eVoLInTHRo wrote: post #35
colinmhayes wrote: post #18

How do we differentiate between econometrics and machine learning? Logistic regression seems like it fits into econometrics better than machine learning to me. There's no regularization. I guess there's gradient descent which can be seen as more machine learning. In the end it's semantics of course, still an interesting distinction.

Econometrics is the application of statistical techniques on economics-related problems, typically to understand relationships between economic phenomena (e.g. income) and things that might be associated with it (e.g. education). Machine learning is typically defined as a way to enable computers to learn from data to accomplish tasks, without explicitly telling them how. Both fields can use logistic regression, regul…

this is a great explanation. Thank you

Re: Logistic regression from scratch

#54

Post by avs733 »

curiousgal wrote: post #27

Very interesting seeing people in the comments debate what is a very basic thing taught in any stats/econometrics class. The idea behind binary regression ( Y is 0 or 1) is that you use a latent variable Y* = beta X + epsilon. X is the matrix of indenpent variables, beta is the vector of coefficients and epsilon is an error term that sums the rest of what X can't explain. Y thus becomes 1 if Y* >0 and 0 otherwise. Se…

so I literally teach statistics to undergraduates...a couple thoughts:

1) agree with others comments that stating with 'very basic' ain't going to work

2) logistic regression actually isn't a common topic in an undergraduate statistics course. I cover it on the second to last day as an 'here is where you can extend it'

3) this isn't a very effective explanation.

4) a big background factor is the misconceptions about statistics that are ingrained in human beings. Learning statistical tests is hard, especially for many engineers and similar STEM types because it is typically the first (and sometimes only) class that deals with concepts of error and variance. Everything else presents an absolute theoretical model. On average, it breaks their brain...

Re: Logistic regression from scratch

#55

Post by thomasahle »

nuclearnice1 wrote: post #38
thomasahle wrote: post #28

Earlier quoted context omitted.

Because Leela (like fastchess mentioned above) has two parts: A neural network predicting good moves, and a tree search exploring the moves suggested and evaluating the resulting positions (with a second net). If the prediction (policy) net had a 100% accuracy, you wouldn't need the tree search part at all.

Got it. Thanks for clarifying. Let me restate. Part one of Leela ranks several chess moves. Part two picks among those. 60% of the time part 2 chooses the #1 ranked move.

That works :-)

One addition: The second part can be run for an arbitrary amount of time, gradually improving the quality of the returned move.

The 60% figure comes from the training games, which are played very quickly, and so don't have a lot of time for refining, thus increasing the prediction accuracy.

In real games, tcec-chess.com/ this "self accuracy" would probably be a bit lower.

Re: Logistic regression from scratch

#56

Post by curiousgal »

ImaCake wrote: post #44
platz wrote: post #39

Earlier quoted context omitted.

fyi, generally when explaining things to non-practitioners, it's a detractor to add qualifiers like - this is a very basic thing - that's all there is to it because although it is a basic thing to you, it's not a basic thing to someone who hasn't spent the same time studying all the concepts beforehand. This is generally why you have to take a class to grok stats rather than just read some reference material and defi…

The author of the parent explains their view well I think (not directed at statistics students). But I would like to make the more general complaint that this attitude seems pervasive in maths. People often fail to include enough detail in their derivations of proofs or their explanations of things and the students are left confused and unsure because they don't see how they got from step 3 to step 4.

I wholeheartedly agree with you, in my case I was typing my comment on my phone while waiting on the train. I wanted to get into as many details as possible but not having access to easily typed math notation made my task harder.

Re: Logistic regression from scratch

#57

Post by vhhn »

cuchoi wrote: post #50
curiousgal wrote: post #41

Earlier quoted context omitted.

From a stats perspective the cutoff is included in the coefficients. If you use a design matrix (add a column of 1s to your variables) you get in a non matrix notation (beta_0 1 + beta_1 X_1 +...) So the threshold can be considered beta_0. In the software, you can get classification models to output class probabilities instead of class labels. You can then use whatever threshold you like for to transform those probab…

The threshold would be beta_0 on every case or only when you have subtracted the mean from your data?

You don't want to demean your dependent (response) binary variable. So you almost always want to keep beta0 to control for any imbalance in your dependent var.

Re: Logistic regression from scratch

#58

Post by jiaaro »

kelvin0 wrote: post #46
curiousgal wrote: post #43

Earlier quoted context omitted.

I totally agree but I was explaining that to practionners, or at least people who consider themselves data scientists. I agree that it is by no means a beginner friendly explanation.

Is that what some people call the 'mansplain' method?

no. all explanations must make some assumption about what readers already know, or else every explanation would begin with teaching english.

also, mansplaining generally refers to assuming people (women, usually) have less pre-existing knowledge than they actually do.

Re: Logistic regression from scratch

#59

Post by nerdponx »

zwaps wrote: post #48
nerdponx wrote: post #8

Earlier quoted context omitted.

It's better to think of linear regression and logistic regression as special cases of the Generalized Linear Model (GLM). In that framework, they are literally the same model with different "settings" - Gaussian vs Bernoulli distribution.

Well, then I am gonna say it's even better to think of linear and logistic regression as special case of M Estimators or GMM. Joke aside, the truth is that logistic regression can be understood based on several assumptions. Above we have the latent variable explanation. Then there is a Bayesian version. There's even a "random utlity" formulation, where one models explicitly choices of an agent with a probabilistic er…

The main benefit of the GLM formulation is the observation that your model implies a particular probability distribution for the target, whether you like it or not. And that your point predictions are in fact conditional means. In my opinion, this is an important aspect of modeling that is glossed over or omitted by a lot of introductory material.

Re: Logistic regression from scratch

#60

Post by cuchoi »

vhhn wrote: post #57
cuchoi wrote: post #50

Earlier quoted context omitted.

The threshold would be beta_0 on every case or only when you have subtracted the mean from your data?

You don't want to demean your dependent (response) binary variable. So you almost always want to keep beta0 to control for any imbalance in your dependent var.

I meant demeaning the independent variables. My understanding is that the beta_0 will have the meaning the curiousgal attach it only if you demean your independent variables.
Post reply on HN