Live data from Hacker News

Visualizing the Math Behind Logistic Regression and Newton's Method

thelaziestprogrammer.com

11–20 of 32 posts

Re: Visualizing the Math Behind Logistic Regression and Newton's Method

#11

The author has far too little mathematical understanding to be teaching anybody (that’s my impression at least). If you don’t understand Newton’s method before reading this, you won’t understand it afterwards. “A method for finding the roots of a polynomial”. Why polynomials? Does it work, always? Is it fast? Why would following the tangent repeatedly be a good idea? “We take the inverse instead of the reciprocal bec…

I am a programmer trying to learn math, so are my intended audience members. That said, I should include facts related to convergence, and maybe even speed compared to SGD. As to the reciprocal -> inverse generalization, do you have any resources you could point we towards to better understand this? Additionally, a concrete answer to "Why would following the tangent repeatedly be a good idea?" has been hard to come b…

In general, it’s not a good idea. And in general, Newton’s method won’t converge.

Newton’s method boils down to replacing your function by a first-order approximation. For a differentiable function, in a small neighbourhood(!), that’s a good approximation (by definition), though, and the zero of the model function will be very close to the zero of the original function (if it lies in that neighbourhood).

PS: i did not expect the poster and author to be the same person, otherwise I would’ve phrased my criticism differently. A SHOW HN would have helped.

PPS: basically the whole reciprocal/inverse confusion only arises because you start the multidimensional case from your iteration formula. If you back to its derivation, and start again from there, you can avoid that.

Re: Visualizing the Math Behind Logistic Regression and Newton's Method

#12

Earlier quoted context omitted.

I am a programmer trying to learn math, so are my intended audience members. That said, I should include facts related to convergence, and maybe even speed compared to SGD. As to the reciprocal -> inverse generalization, do you have any resources you could point we towards to better understand this? Additionally, a concrete answer to "Why would following the tangent repeatedly be a good idea?" has been hard to come b…

In general, it’s not a good idea. And in general, Newton’s method won’t converge. Newton’s method boils down to replacing your function by a first-order approximation. For a differentiable function, in a small neighbourhood(!), that’s a good approximation (by definition), though, and the zero of the model function will be very close to the zero of the original function (if it lies in that neighbourhood). PS: i did no…

> In general, it’s not a good idea. And in general, Newton’s method won’t converge.

Right, but this blog post isn't about the general case of using Newton's method to find roots, it's about using Newton's method for solving logistic regression for which it is perfectly suited, though there are better methods as well, of course.

Re: Visualizing the Math Behind Logistic Regression and Newton's Method

#13
Since the author is reading, a few small typos, followed by one slightly more substantial comment: 'simgoid' should be 'sigmoid' (S-shaped); `x y = log(x) + log(y)` should be `log(x y) = log(x) + log(y)`;'guarentee' should be 'guarantee'; 'recipricol' should be 'reciprocal'.

I would like to see some mention of the fact that the division by the gradient is a meaningless, purely formal motivation for the correct step (inverting the Hessian) that follows.

Re: Visualizing the Math Behind Logistic Regression and Newton's Method

#15
post #14

For the graphs of the home price / bathroom data set, what does the vertical axis represent? I don't see it labeled or discussed anywhere.

I just took the index of the data point (i.e. 1, 250) as the y-axis with the intention of "Stretching out" the data set along the y-axis. Otherwise the data would otherwise be illegibly compressed on a 1-D number line.

In retrospect, I could have better represented the data with 2 overlying histograms, but this (somewhat) captures the intent of showing that "more expensive houses tend to have more than 2 bathrooms".

Re: Visualizing the Math Behind Logistic Regression and Newton's Method

#17
post #13

Since the author is reading, a few small typos, followed by one slightly more substantial comment: 'simgoid' should be 'sigmoid' (S-shaped); `x y = log(x) + log(y)` should be `log(x y) = log(x) + log(y)`;'guarentee' should be 'guarantee'; 'recipricol' should be 'reciprocal'. I would like to see some mention of the fact that the division by the gradient is a meaningless, purely formal motivation for the correct step (…

Thanks for the feedback, I co-worker just jabbed me with regarding the log property mistake also...

As to the motivation for the correct step: can you point me to a resource that explains this? Not sure I follow...

Re: Visualizing the Math Behind Logistic Regression and Newton's Method

#18
post #13

Since the author is reading, a few small typos, followed by one slightly more substantial comment: 'simgoid' should be 'sigmoid' (S-shaped); `x y = log(x) + log(y)` should be `log(x y) = log(x) + log(y)`;'guarentee' should be 'guarantee'; 'recipricol' should be 'reciprocal'. I would like to see some mention of the fact that the division by the gradient is a meaningless, purely formal motivation for the correct step (…

Thanks for the feedback, I co-worker just jabbed me with regarding the log property mistake also... As to the motivation for the correct step: can you point me to a resource that explains this? Not sure I follow...

> As to the motivation for the correct step: can you point me to a resource that explains this? Not sure I follow...

You write an equation involving division by the gradient. This is an illegal operation (one cannot divide by a vector), and your final recipe doesn't do it. As far as I can tell, you are writing down the incorrect, illegally-vector-inverting formula as motivation for the correct formula involving the (inverse of the) Hessian. All I am suggesting is that you say explicitly something like "Of course, this formula as written is not literally correct; one cannot actually divide by a vector. The correct procedure is explained below."

(Incidentally, speaking of inverses, another poster (https://news.ycombinator.com/item?id=14881265) has mentioned that it may be a bit confusing to speak of the inverse of a matrix rather than the reciprocal, since (as I interpret that other poster's point) the reciprocal of a matrix is just its inverse. I might prefer to say something like "We write $H_{\ell(\theta)}^{-1}\nabla\ell(\theta)$ rather than $\frac{\nabla\ell(\theta)}{H_\ell(\theta)}$ to emphasise that we are inverting a matrix, not a scalar, so that the order of multiplication matters.")

Re: Visualizing the Math Behind Logistic Regression and Newton's Method

#20

I search everywhere about the difference between Newton's Method and Gradient Descent? but I couldn't find something that useful. Can u suggest any website/ article where I can learn the difference?

The comment right below has 2 links:

Gradient Descent explored similarly to this post: http://thelaziestprogrammer.com/sharrington/math-of-machine-...

The difference between the two: https://www.quora.com/In-optimization-why-is-Newtons-method-...

Post reply on HN