Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Matrix Calculus for Deep Learning

explained.ai

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Matrix Calculus for Deep Learning

#31

Post by nabla9 »

madenine wrote: post #2

I’m in favor of there being more and better resources to learn anything out there, but every time I see a deep learning 101 type material all I can think is “who is this for?”. In ~July 2016 I was at a presentation by NVidia at GW in DC. They showed off how easy it was to build out and train a model using some of their tooling (Digits maybe?). After the demo they opened it up for questions and a grad student ‘asked’…

> So while this looks like a great resource - who’s it for?

I give you an analogy. Electricity. Who needs to know complex numbers and differential equations to understand electricity? Technician, civil engineer, scientist or research engineer?

Technician who just wires the house don't need math. They just read the wiring instructions and follow standard practices. Nvidia boasts about the tools it builds for 'ML technicians' in this analogy.

You need to know math if you are building new architectures and applying complex models for something nontrivial. It's not going to work first time and you need to know what's going on. Even if you are the 'civil engineer' in this analogy you should be able to read the math and understand it even if you don't do the math by yourself. You won't be able to do literary research and learn new stuff if you can't read math fluently.

If you are programmer who is given ML tools to implement something someone else designed and understand you don't need this or use existing models, you don't need this. Your career might benefit from knowing it but you can manage without.

Re: Matrix Calculus for Deep Learning

#32

Post by p1esk »

nabla9 wrote: post #31
madenine wrote: post #2

I’m in favor of there being more and better resources to learn anything out there, but every time I see a deep learning 101 type material all I can think is “who is this for?”. In ~July 2016 I was at a presentation by NVidia at GW in DC. They showed off how easy it was to build out and train a model using some of their tooling (Digits maybe?). After the demo they opened it up for questions and a grad student ‘asked’…

> So while this looks like a great resource - who’s it for? I give you an analogy. Electricity. Who needs to know complex numbers and differential equations to understand electricity? Technician, civil engineer, scientist or research engineer? Technician who just wires the house don't need math. They just read the wiring instructions and follow standard practices. Nvidia boasts about the tools it builds for 'ML techn…

I believe the OP's point was that the math described in the article is too simple and not enough to do any serious research. Anyone who attempts to do NN research already knows this material (and a lot more). This tutorial could be useful to someone who wanted to implement simple backprop from scratch, but all DL libraries already do it automatically. Someone who just wants to learn a bit about NNs to classify images or generate text does not need to know this, and someone who wants to make a breakthrough in NN theory already knows it. So yes it's not very clear who is the target audience here. I'm guessing it's for a bright highschooler who just learned calculus and who is interested in how NNs work. For such students I'd recommend reading http://neuralnetworksanddeeplearning.com instead.

Re: Matrix Calculus for Deep Learning

#33

Post by improbable22 »

fg6hr wrote: post #27
publicrootkey wrote: post #14

Earlier quoted context omitted.

If I understand your question correctly, this is exactly what Bayes' Theorem deals with. Namely that B must be true within some probability given that B is true represented as P(A | B).

Let's consider a simple image classifier: 64x64 grayscale input, 10x1 output that detects 10 classes of images. The model: y=tanh(Ax+b). You'd probably say that there is no way this will work because this model is too simple. But can you explain why this won't work? Can you tell what the maximum accuracy this model can reach? What kind of datasets this model would work better on? [Edit] Whether this model works is qu…

I think you're asking for the holy grail here. Everyone would love to have such a thing but nobody thinks it's likely to be possible.

So they settle for much smaller targets. Either of understanding how much simpler systems work. Or of trying to understand a little bit the effect of tweaking something in some more complicated model.

Perhaps you should think of these two approaches as analogous to doing simple chemistry (what shape is a sugar molecule? A DNA molecule?) vs trying out drugs (if you eat the bark of this tree, you don't get malaria! Let's refine that stuff). Both can be useful, but they are very far from a unified theory of how your body works.

Re: Matrix Calculus for Deep Learning

#34

Post by p1esk »

SpaceManNabs wrote: post #18
nightcracker wrote: post #16

Earlier quoted context omitted.

There is a subfield that does serious mathematics, but their results are usually far removed from the state of the art stuff. Their results usually look like "a neural network with 1 hidden layer is a universal approximator" or "exponential expected convergence speed on linear relations for ".

Not necessarily true. The polynomial time for the escape of saddle points by using stochastic noise is pretty practical.

How in the world is this "pretty practical"? It would be if people used this theorem to come up with an idea of SGD, but that's not what happened. SGD appeared as a way to overcome the practical constraint of computing full GD. Not to mention that "polynomial time" is meaningless to any practitioner.

Re: Matrix Calculus for Deep Learning

#35

Post by chestervonwinch »

p1esk wrote: post #30
jeeceebees wrote: post #19

Earlier quoted context omitted.

Of course there is. All the building blocks that people are mix and matching in networks nowadays were introduced at some point. The paper that introduced batch norm, adaptive instance norm, attention heads, or any module used in a network have an extensive discussion of the motivation for their existance, some derivation or proof that they do what you want, and an empirical test to show it helps in practice. The rea…

nobody really knows how AI works is one of those myths told by the media It's not a myth. No one really understands how neural networks work. We don't know why a particular model works well. Or why any model works well. For example no one can answer why NNs generalize so well even when they have enough learning capacity to memorize all training examples. We can guess, but we don't know for sure. Most of the proofs yo…

> NNs are currently at a similar point as where physics was before Newton and before calculus.

I'm more inclined to compare with the era after Newton and Leibniz, but prior to the development of rigorous analysis. If you look at this time period, the analogy fits a bit better IMO -- you have a proliferation of people using calculus techniques to great advantage for solving practical problems, but no real foundations propping the whole thing up (e.g., no definition of a limit, continuity, notions of how to deal with infinite series, etc.).

Re: Matrix Calculus for Deep Learning

#36

Post by stinky »

madenine wrote: post #2

I’m in favor of there being more and better resources to learn anything out there, but every time I see a deep learning 101 type material all I can think is “who is this for?”. In ~July 2016 I was at a presentation by NVidia at GW in DC. They showed off how easy it was to build out and train a model using some of their tooling (Digits maybe?). After the demo they opened it up for questions and a grad student ‘asked’…

> So it’s for people that have been getting into DL but also haven’t bothered or needed to look up the math concepts?

Yes.

Re: Matrix Calculus for Deep Learning

#37

Post by joshvm »

rosstex wrote: post #12
madenine wrote: post #2

I’m in favor of there being more and better resources to learn anything out there, but every time I see a deep learning 101 type material all I can think is “who is this for?”. In ~July 2016 I was at a presentation by NVidia at GW in DC. They showed off how easy it was to build out and train a model using some of their tooling (Digits maybe?). After the demo they opened it up for questions and a grad student ‘asked’…

I don't even know what course you'd learn matrix calculus in, but it was a necessity for my upper level ML courses. This website would have been a godsend, and would have spared TAs many hours figuring out what knowledge we were missing. We got by with the Wikipedia page...

High level computer science courses straddle several disciplines and you end up with weird stuff like computer vision being the purview of the electrical engineering department. EEs tend to know some of this stuff because a lot of matrix algebra comes up in control/optimisation theory.

In physics we did matrix calculus primarily for electromagnetism and fluid dynamics. Maxwell's equations are the first time most students see the div/curl operator and it's also used in e.g. Navier-Stokes. But even though we were taught it, I don't think we really bothered to remember what a "Jacobian" is.

A lot of this stuff also comes up in physical rendering.

Re: Matrix Calculus for Deep Learning

#38

Post by sjg007 »

nabla9 wrote: post #31
madenine wrote: post #2

I’m in favor of there being more and better resources to learn anything out there, but every time I see a deep learning 101 type material all I can think is “who is this for?”. In ~July 2016 I was at a presentation by NVidia at GW in DC. They showed off how easy it was to build out and train a model using some of their tooling (Digits maybe?). After the demo they opened it up for questions and a grad student ‘asked’…

> So while this looks like a great resource - who’s it for? I give you an analogy. Electricity. Who needs to know complex numbers and differential equations to understand electricity? Technician, civil engineer, scientist or research engineer? Technician who just wires the house don't need math. They just read the wiring instructions and follow standard practices. Nvidia boasts about the tools it builds for 'ML techn…

I don't think ML and NNs are at the point yet where you don't need to understand the math.

Re: Matrix Calculus for Deep Learning

#39

Post by mjburgess »

madenine wrote: post #2

I’m in favor of there being more and better resources to learn anything out there, but every time I see a deep learning 101 type material all I can think is “who is this for?”. In ~July 2016 I was at a presentation by NVidia at GW in DC. They showed off how easy it was to build out and train a model using some of their tooling (Digits maybe?). After the demo they opened it up for questions and a grad student ‘asked’…

This is a very unempahtetic, almost anti-educational, comment.

Pick any journey to any destination. This article occurs at many points along them.

Need to have a thorough understanding of math? Then this is a starting point.

Don't? then this is an endpoint.

Re: Matrix Calculus for Deep Learning

#40

Post by tel »

fg6hr wrote: post #13

A genuine question: is there any math behind ML at all? For example, is there any solid theory, with proven theorems, that would tell us what happens when we add another conv layer here or use a 3x3 conv kernel instead of a 2x2 one over there, or replace that tanh with a relu? From my limited understanding, ML roughly works like this: we shuffle around the ML graph, using some intuition, off-load it to a cluster of G…

First up, what you seem to be talking about is Deep Learning, not Machine Learning in general. In more general ML there are many theorems, some also apply to DL.

Also, the step of "shuffle around the ML graph using some intuition" involves gathering that intuition, which usually arises from a great deal of mathematical competence. A 3x3 conv kernel versus a 2x2 one can, for instance, be discussed in terms of Fourier theory and mathematical image processing, but areas with huge built-in theory.

Things like replacing the activation function were initially studied anecdotally. People realized that in some settings one activation function or another would lead. Eventually, there was also theory showing that in large nets of stable configurations, there was serious interaction between the initialization method and the activation function and problems like poor backprop signal propagation were tackled theoretically and practically.

Generally, the mystery comes from the vast parameterization of these DL models. They operate in a space that's very hard to generalize—large, finite spaces. Small finite spaces get treated exhaustively. Infinite spaces get treated asymptotically. Large finite spaces get bounded on either side by those methods.

So yes, there might feel like there's a dearth of theory in DL when it comes to the large scale behavior of a general network. That can be super frustrating. At the same time, people are trying to push through and create more theory every day.

Post reply on HN