Visualizing the Math Behind Logistic Regression and Newton's Method
thelaziestprogrammer.com
Visualizing the Math Behind Logistic Regression and Newton's Method
1–10 of 32 posts
Re: Visualizing the Math Behind Logistic Regression and Newton's Method
#2What is the difference between Newton's Method and Gradient Descent?
Edit: Found an answer: https://www.quora.com/In-optimization-why-is-Newtons-method-...
Re: Visualizing the Math Behind Logistic Regression and Newton's Method
#3What is the difference between Newton's Method and Gradient Descent? Edit: Found an answer: https://www.quora.com/In-optimization-why-is-Newtons-method-...
I wrote this post (first) on Gradient Descent that will be helpful if you liked the OP.
http://thelaziestprogrammer.com/sharrington/math-of-machine-...
This looks at the algorithm the same way that the OP does (mathematically, visually, programmatically), only it applies it to a simpler Linear Regression model.
Re: Visualizing the Math Behind Logistic Regression and Newton's Method
#4This is a really nice introduction to logistic regression, well done! My one quibble with the OP is the jump into Newton's method. Maybe a derivation to explain the five steps would help. Thanks!
Re: Visualizing the Math Behind Logistic Regression and Newton's Method
#5This is a really nice introduction to logistic regression, well done! My one quibble with the OP is the jump into Newton's method. Maybe a derivation to explain the five steps would help. Thanks!
Thank you. I err'ed on the side of caution to (try) to reduce information overload, but I also didn't want to assume too much prior knowledge. Delicate balance...
Maybe a linked post that dives deeper into the 5 steps of Newton's Method? Would that be more approachable?
Re: Visualizing the Math Behind Logistic Regression and Newton's Method
#6This is a really nice introduction to logistic regression, well done! My one quibble with the OP is the jump into Newton's method. Maybe a derivation to explain the five steps would help. Thanks!
Thank you. I err'ed on the side of caution to (try) to reduce information overload, but I also didn't want to assume too much prior knowledge. Delicate balance... Maybe a linked post that dives deeper into the 5 steps of Newton's Method? Would that be more approachable?
Yeah, that would work. Honestly, when I studied logistic regression, we used gradient descent, so seeing it with Newton's method was the draw for me. I can go find the resources myself of course, but since I was reading the article I thought it would be convenient there :)
Re: Visualizing the Math Behind Logistic Regression and Newton's Method
#7The author has far too little mathematical understanding to be teaching anybody (that’s my impression at least). If you don’t understand Newton’s method before reading this, you won’t understand it afterwards. “A method for finding the roots of a polynomial”. Why polynomials? Does it work, always? Is it fast? Why would following the tangent repeatedly be a good idea? “We take the inverse instead of the reciprocal because it’s a matrix”. Not impressed.
Re: Visualizing the Math Behind Logistic Regression and Newton's Method
#8The author has far too little mathematical understanding to be teaching anybody (that’s my impression at least). If you don’t understand Newton’s method before reading this, you won’t understand it afterwards. “A method for finding the roots of a polynomial”. Why polynomials? Does it work, always? Is it fast? Why would following the tangent repeatedly be a good idea? “We take the inverse instead of the reciprocal bec…
To elaborate a bit: Newton’s method is amazing. But it’s also dangerous and difficult to use. You can talk about it for a long time and use it as the centrepiece of an course on vector analysis. This treatment does not explain anything - it might as well just say “we use the magical newton method, a discussion of which is beyond the scope of this work”. I’d find that honest.
Re: Visualizing the Math Behind Logistic Regression and Newton's Method
#9The author has far too little mathematical understanding to be teaching anybody (that’s my impression at least). If you don’t understand Newton’s method before reading this, you won’t understand it afterwards. “A method for finding the roots of a polynomial”. Why polynomials? Does it work, always? Is it fast? Why would following the tangent repeatedly be a good idea? “We take the inverse instead of the reciprocal bec…
Back when I was in high school Newton's Method was typically introduced as a method to find roots of a polynomial, usually in "Algebra 2" or Trigonometry or about then. That's because there is an easy to grasp formula for it that can be used in the context of a very familiar mathematical construct (polynomials) and produced from differentiation that doesn't technically require an understanding of calc to use (as opposed to actually understand, derive, etc.). I still use it pre-calc when I tutor younger students because it's a great exploratory tool.
Re: Visualizing the Math Behind Logistic Regression and Newton's Method
#10The author has far too little mathematical understanding to be teaching anybody (that’s my impression at least). If you don’t understand Newton’s method before reading this, you won’t understand it afterwards. “A method for finding the roots of a polynomial”. Why polynomials? Does it work, always? Is it fast? Why would following the tangent repeatedly be a good idea? “We take the inverse instead of the reciprocal bec…
I am a programmer trying to learn math, so are my intended audience members.
That said, I should include facts related to convergence, and maybe even speed compared to SGD.
As to the reciprocal -> inverse generalization, do you have any resources you could point we towards to better understand this?
Additionally, a concrete answer to "Why would following the tangent repeatedly be a good idea?" has been hard to come by for me. I am able to visualize this, but if you have resources that explain this well please share.