Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Matrix Calculus for Deep Learning

explained.ai

Post reply on HN
Print view View original

41–49 of 49 posts

Re: Matrix Calculus for Deep Learning

#41

Post by justinmeiners »

olooney wrote: post #11
madenine wrote: post #2

I’m in favor of there being more and better resources to learn anything out there, but every time I see a deep learning 101 type material all I can think is “who is this for?”. In ~July 2016 I was at a presentation by NVidia at GW in DC. They showed off how easy it was to build out and train a model using some of their tooling (Digits maybe?). After the demo they opened it up for questions and a grad student ‘asked’…

> So while this looks like a great resource - Who's it for? Undergraduates, or graduate students who didn't happen to take the right prerequisites. Most STEM degrees require vector calculus, but few require matrix calculus. A physics undergrad might see matrix calculus if they studied general relativity, or math undergrad interested in optimization or differential geometry. A statistics major might have seen it when…

> Most STEM degrees require vector calculus, but few require matrix calculus

Gradients, Jacobians, etc are typically covered in a multivariable calculus class along with vector calculus (line integrals, Green's theorem, Stoke's theorem, etc). This is required for engineering and physics degrees.

Re: Matrix Calculus for Deep Learning

#42

Post by JamesBarney »

madenine wrote: post #2

I’m in favor of there being more and better resources to learn anything out there, but every time I see a deep learning 101 type material all I can think is “who is this for?”. In ~July 2016 I was at a presentation by NVidia at GW in DC. They showed off how easy it was to build out and train a model using some of their tooling (Digits maybe?). After the demo they opened it up for questions and a grad student ‘asked’…

My guess is it's for people who have gotten something working in the past and are now looking to go a little deeper.

Re: Matrix Calculus for Deep Learning

#43

Post by fg6hr »

improbable22 wrote: post #33
fg6hr wrote: post #27

Earlier quoted context omitted.

Let's consider a simple image classifier: 64x64 grayscale input, 10x1 output that detects 10 classes of images. The model: y=tanh(Ax+b). You'd probably say that there is no way this will work because this model is too simple. But can you explain why this won't work? Can you tell what the maximum accuracy this model can reach? What kind of datasets this model would work better on? [Edit] Whether this model works is qu…

I think you're asking for the holy grail here. Everyone would love to have such a thing but nobody thinks it's likely to be possible. So they settle for much smaller targets. Either of understanding how much simpler systems work. Or of trying to understand a little bit the effect of tweaking something in some more complicated model. Perhaps you should think of these two approaches as analogous to doing simple chemist…

ML is more like alchemy, I'd say: mixing components using intuition and experience, but without understanding what these components really are and why they work. In this analogy, AI is the recipe to make gold and the ML alchemists haven't invented nuclear physics yet.

Re: Matrix Calculus for Deep Learning

#44

Post by p1esk »

chestervonwinch wrote: post #35
p1esk wrote: post #30

Earlier quoted context omitted.

nobody really knows how AI works is one of those myths told by the media It's not a myth. No one really understands how neural networks work. We don't know why a particular model works well. Or why any model works well. For example no one can answer why NNs generalize so well even when they have enough learning capacity to memorize all training examples. We can guess, but we don't know for sure. Most of the proofs yo…

> NNs are currently at a similar point as where physics was before Newton and before calculus. I'm more inclined to compare with the era after Newton and Leibniz, but prior to the development of rigorous analysis. If you look at this time period, the analogy fits a bit better IMO -- you have a proliferation of people using calculus techniques to great advantage for solving practical problems, but no real foundations…

Maybe. On the other hand, maybe a rigorous mathematical analysis of NNs is as useful as a rigorous mathematical analysis of computer architectures - not very useful. Maybe all you need is just to keep scaling it up, adding some clever optimizations in the process (none of the great CPU ideas like caches, pipelining, out of order execution, branch prediction, etc came from rigorous mathematical analysis).

Or maybe it's as useful as a rigorous mathematical analysis of a brain - again, not very useful, because for us (people who develop AI systems), it would be far more valuable to understand a brain on a circuit level, or an architecture level, rather than on a mathematical theory level. The latter would be interesting, but probably too complex to be useful, while the former would most likely lead to dramatic breakthroughs in terms of performance and capabilities of the AI systems.

So maybe we just need to keep doing what we have been doing in DL field in the last 10 years - trying/revisiting various ideas, scaling them up, and evolving the architectures the same way we've been evolving our computers for the last 100 years, with the hope there will be more clues from neuroscience. I think we just need more ideas like transformers, capsules, or neural Turing machines, and computers that are getting ~20% faster every year.

Re: Matrix Calculus for Deep Learning

#45

Post by fg6hr »

improbable22 wrote: post #33
fg6hr wrote: post #27

Earlier quoted context omitted.

Let's consider a simple image classifier: 64x64 grayscale input, 10x1 output that detects 10 classes of images. The model: y=tanh(Ax+b). You'd probably say that there is no way this will work because this model is too simple. But can you explain why this won't work? Can you tell what the maximum accuracy this model can reach? What kind of datasets this model would work better on? [Edit] Whether this model works is qu…

I think you're asking for the holy grail here. Everyone would love to have such a thing but nobody thinks it's likely to be possible. So they settle for much smaller targets. Either of understanding how much simpler systems work. Or of trying to understand a little bit the effect of tweaking something in some more complicated model. Perhaps you should think of these two approaches as analogous to doing simple chemist…

My guess is that what I'm asking for isn't that complex and could be done by a few serious mathematicians in a few years. The dynamics of tanh(Ax+b) is hardly more complex than Naiver-Stokes equation or the modern topology theory.

Re: Matrix Calculus for Deep Learning

#46

Post by hgoel »

p1esk wrote: post #32
nabla9 wrote: post #31

Earlier quoted context omitted.

> So while this looks like a great resource - who’s it for? I give you an analogy. Electricity. Who needs to know complex numbers and differential equations to understand electricity? Technician, civil engineer, scientist or research engineer? Technician who just wires the house don't need math. They just read the wiring instructions and follow standard practices. Nvidia boasts about the tools it builds for 'ML techn…

I believe the OP's point was that the math described in the article is too simple and not enough to do any serious research. Anyone who attempts to do NN research already knows this material (and a lot more). This tutorial could be useful to someone who wanted to implement simple backprop from scratch, but all DL libraries already do it automatically. Someone who just wants to learn a bit about NNs to classify images…

But someone who wants to contribute to the research doesn't just have this knowledge pop into their mind out of nowhere. They're going to learn it from somewhere and what's wrong with one more resource to help out with that.

Re: Matrix Calculus for Deep Learning

#47

Post by improbable22 »

fg6hr wrote: post #45
improbable22 wrote: post #33

Earlier quoted context omitted.

I think you're asking for the holy grail here. Everyone would love to have such a thing but nobody thinks it's likely to be possible. So they settle for much smaller targets. Either of understanding how much simpler systems work. Or of trying to understand a little bit the effect of tweaking something in some more complicated model. Perhaps you should think of these two approaches as analogous to doing simple chemist…

My guess is that what I'm asking for isn't that complex and could be done by a few serious mathematicians in a few years. The dynamics of tanh(Ax+b) is hardly more complex than Naiver-Stokes equation or the modern topology theory.

tanh(Ax+b) is simple, but the dataset it's supposed to work on is not easily summarised. I think that's the huge difference. The guys doing "sugar molecule" studies make progress by taking much simpler datasets, like random points.

Naiver-Stokes is much simpler because it operates by itself. Of course turbulence is hard but even there we usually care about its coarse features, we'd be content to throw away almost all the information provided the calculation of the wing's lift works out OK.

Re: Matrix Calculus for Deep Learning

#48

Post by improbable22 »

fg6hr wrote: post #43
improbable22 wrote: post #33

Earlier quoted context omitted.

I think you're asking for the holy grail here. Everyone would love to have such a thing but nobody thinks it's likely to be possible. So they settle for much smaller targets. Either of understanding how much simpler systems work. Or of trying to understand a little bit the effect of tweaking something in some more complicated model. Perhaps you should think of these two approaches as analogous to doing simple chemist…

ML is more like alchemy, I'd say: mixing components using intuition and experience, but without understanding what these components really are and why they work. In this analogy, AI is the recipe to make gold and the ML alchemists haven't invented nuclear physics yet.

But now we know there was physics at the bottom of alchemy. Whereas demonology at best leads you to psychiatry, and we still don't have simple models of what works there. Nor much hope of finding them. Thinking is a messy business.

Re: Matrix Calculus for Deep Learning

#49

Post by SpaceManNabs »

p1esk wrote: post #34
SpaceManNabs wrote: post #18

Earlier quoted context omitted.

Not necessarily true. The polynomial time for the escape of saddle points by using stochastic noise is pretty practical.

How in the world is this "pretty practical"? It would be if people used this theorem to come up with an idea of SGD, but that's not what happened. SGD appeared as a way to overcome the practical constraint of computing full GD. Not to mention that "polynomial time" is meaningless to any practitioner.

it did come with a new online update for orthogonal tensor decomposition using higher order moments and with comments on NP-hardness for 4th order and higher.

In addition, it came with tricks with how much noise to inject in certain situations. "How much noise do you need is enough to escape?" which is pretty practical

Post reply on HN