Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Matrix Calculus You Need for Deep Learning

explained.ai

Post reply on HN
Print view View original

31–40 of 79 posts

Re: The Matrix Calculus You Need for Deep Learning

#31

Post by pX0r »

roel_v wrote: post #2

So I have a question somewhat related to this that I never knew where/who to ask (well actually I asked a few mathematicians at a university I work with whose answers I couldn't understand - their answers were almost as impenetrable as the Wikipedia page, and some engineering scientists who I thought would be more into 'applied math' but they didn't know. So I'm hoping some data science people reading this would bett…

A logistic regression model is typically used for a classification task. 'Fitting a logistic model' entails finding optimal coefficients / 'weights' of input features such that classification error is minimised.

For a binary classification task, one could simply calculate mean squared error between predicted values and actual labels (as in linear regression) and then proceed to find the optimal weights iteratively using gradient descent. But the sigmoid shape of the logistic function makes gradient descent a poor choice of an optimization technique (w.r.t. lack of guarantee of finding a global optimum).

A surer way to find globally optimal weights is using the Newton's method of calculating weight updates. This is a numerical optimization technique that requires one to calculate the 1st and 2nd order derivatives of the error function. The matrix that 'calculates' the 1st order derivative is called a Jacobian and the one that calculates the 2nd order derivative is called a Hessian...

Re: The Matrix Calculus You Need for Deep Learning

#32

Post by mxwsn »

mlevental wrote: post #30
TJSomething wrote: post #9

Earlier quoted context omitted.

Those Wikipedia pages are kind of awful for pedagogy, but they have the right equations, so I won't cover those. Say we have a curve that corresponds to how good of a fit your model is. We want to try to find the maximum on that curve. However, calculating every point of the curve is too expensive, so we want to minimize the number of points we have to check. So, we start with a guess as to the highest point on the c…

what is the name of the first method (fit parabolic surface)?

In optimization it's known as Newton's method.

See the third section here for an intuitive image of repeated parabola-fitting. https://ardianumam.wordpress.com/2017/09/27/newtons-method-o...

Wiki: https://en.wikipedia.org/wiki/Newton%27s_method_in_optimizat...

Re: The Matrix Calculus You Need for Deep Learning

#33

Post by closetCS »

tw1010 wrote: post #19
luk32 wrote: post #14

If someone likes more lecture style explanation I can recommend 3blue1brown's material on YouTube. He explained in a pretty good an accessible way imho. I didn't learn artificial neural network stuff from there. I knew those concepts but I didn't know the matrix formalism applied to it. So this was really nice to understand why GPUs are good for this. Math-wise it was really nice watch.

It's amazing what a rich-get-richer effect products and content that really manage to solve problems in a high quality way get in comments sections around the web. (E.g. 3B1B.)

Is that such a bad thing?

The web has allowed the sharing of high quality content across the world for little to no cost, but has also created so much noise for the average user that they have little to no hope of finding the high quality content on their own. Comment sections across the web fix this problem by promoting producers that offer a superior product. This encourages everyone to make better content.

Re: The Matrix Calculus You Need for Deep Learning

#35

Post by pvg »

Myrmornis wrote: post #29
ghaff wrote: post #25

Earlier quoted context omitted.

Unfortunately, mathematics is one of the areas where Wikipedia is pretty awful in general. The articles seem mostly written for people who pretty much already understand the topic in question. Of course, you always have to assume some knowledge base but the stereotypical jargon-filled Wilipedia approach is particularly off-putting in this area.

I understand what you’re trying to say, but Wikipedia is a fantastic resource for mathematics. “Pretty awful” is not a correct choice of words. But yes, much of it is written at beyond-undergrad-math level. And undergrad math is already advanced! And no I’m not someone with a math PhD talking down! I’m struggling through teaching myself undergrad math.

The only way "pretty awful" is incorrect is that it is too polite and reserved. Reams upon reams of pages are written completely at odds with Wikipedia's own style guidelines and common-sense expectations of what one might find in an encyclopedia. Unlike some famously dense mathematical texts, wikipedia maths pages don't even come with any of the benefits of brevity or focus. It's like a giant joke competition of who can describe every trivial thing in the most abstract and abstruse way except it got out of hand and the participants forgot it was supposed to be a joke. Mathworld and similar sites will help you much more with undergrad maths.

Re: The Matrix Calculus You Need for Deep Learning

#36

Post by shrimp_emoji »

tw1010 wrote: post #19
luk32 wrote: post #14

If someone likes more lecture style explanation I can recommend 3blue1brown's material on YouTube. He explained in a pretty good an accessible way imho. I didn't learn artificial neural network stuff from there. I knew those concepts but I didn't know the matrix formalism applied to it. So this was really nice to understand why GPUs are good for this. Math-wise it was really nice watch.

It's amazing what a rich-get-richer effect products and content that really manage to solve problems in a high quality way get in comments sections around the web. (E.g. 3B1B.)

Yep. And 3b1b seems to agree: https://youtu.be/VcgJro0sTiM

Re: The Matrix Calculus You Need for Deep Learning

#37

Post by hypersoar »

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

Re: The Matrix Calculus You Need for Deep Learning

#38

Post by Iwan-Zotow »

hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

Sure, Goodfellow book

http://www.deeplearningbook.org/

Re: The Matrix Calculus You Need for Deep Learning

#39

Post by Twisol »

tw1010 wrote: post #19
luk32 wrote: post #14

If someone likes more lecture style explanation I can recommend 3blue1brown's material on YouTube. He explained in a pretty good an accessible way imho. I didn't learn artificial neural network stuff from there. I knew those concepts but I didn't know the matrix formalism applied to it. So this was really nice to understand why GPUs are good for this. Math-wise it was really nice watch.

It's amazing what a rich-get-richer effect products and content that really manage to solve problems in a high quality way get in comments sections around the web. (E.g. 3B1B.)

I'm not really sure why you're being downvoted. You're right that 3B1B really does manage to solve a problem in a high-quality way, and it's amazing how much of an effect that really has on people, especially considering the relatively niche topics that 3B1B goes into. (You'd think that his "Essence of" series would be more popular, but the one-off problem analyses have ridiculously more views in general.)

In some ways, it is a "rich-get-richer" effect. But creators like 3B1B expend a lot of time and resources to do what they do, and the word of mouth he gets is an acknowledgement that the work he does is worth the money and views we provide.

Re: The Matrix Calculus You Need for Deep Learning

#40

Post by mlthoughts2018 »

hypersoar wrote: post #37

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

I made the same transition earlier in my career. One book on deep learning that meets your requirements is [0]. It’s readable, covers a broad set of modern topics, and has pragmatic tips for real use cases.

For general machine learning, there are many, many books. A good intro is [1] and a more comprehensive, reference sort of book is [2]. Frankly, by this point, even reading the documentation and user guide of scikit-learn has a fairly good mathematical presentation of many algorithms. Another good reference book is [3].

Finally, I would also recommend supplementing some of that stuff with Bayesian analysis, which can address many of the same problems, or be intermixed with machine learning algorithms, but which is important for a lot of other reasons too (MCMC sampling, hierarchical regression, small data problems). For that I would recommend [4] and [5].

Stay away from bootcamps or books or lectures that seem overly branded with “data science.” This usually means more focus on data pipeline tooling, data cleaning, shallow details about a specific software package, and side tasks like wrapping something in a webservice.

That stuff is extremely easy to learn on the job and usually needs to be tailored differently for every different project or employer, so it’s a relative waste of time unless it is the only way you can get a job.

[0]: https://www.amazon.com/Deep-Learning-Adaptive-Computation-Ma... >

[1]: https://www.amazon.com/Pattern-Classification-Pt-1-Richard-D... >

[2]: https://www.amazon.com/Pattern-Recognition-Learning-Informat... >

[3]: http://www.web.stanford.edu/~hastie/ElemStatLearn/ >

[4]: http://www.stat.columbia.edu/~gelman/book/ >

[5]: http://www.stat.columbia.edu/~gelman/arm/ >

Post reply on HN