If anybody want to read the real stuff then here is the reference:
Matrix Computations https://www.cs.cornell.edu/cv/Books/GVL/index.htm
Matrix Calculus for Deep Learning
21–30 of 49 posts
Re: Matrix Calculus for Deep Learning
#22A genuine question: is there any math behind ML at all? For example, is there any solid theory, with proven theorems, that would tell us what happens when we add another conv layer here or use a 3x3 conv kernel instead of a 2x2 one over there, or replace that tanh with a relu? From my limited understanding, ML roughly works like this: we shuffle around the ML graph, using some intuition, off-load it to a cluster of G…
Of course there is. All the building blocks that people are mix and matching in networks nowadays were introduced at some point. The paper that introduced batch norm, adaptive instance norm, attention heads, or any module used in a network have an extensive discussion of the motivation for their existance, some derivation or proof that they do what you want, and an empirical test to show it helps in practice. The rea…
Actually, I read the batch norm paper and maybe I forgot important details, but it roughly went like this: "here we add a term `b` to make sure the mean values of Ax+b are zero and that will help us with convergence; ah, and here is a covariance matrix!", but no quantitative proofs about how much that convergence was helped. No, I intuitively agree that shifting the mean value to zero should help, but math taught me that there is a huge difference between a seemingly correct statement and its proof. The ML papers seem to just state these seemingly correct ideas without real, proof-backed understanding, why this works. In other words, ML is entirely about empirical results, peppered with math-like terminology. But don't take my blunt writing style personally.
Let's take the simplest example: recognizing the grayscale 30x80 pictures with 0-9 digits. IIRC, this is called the MNIST example and can be done by my cat in 1 hour without prior knowledge. Let's choose the probably simplest model: 2400 inputs are fully connected with a 1024 vector that's fully connected with a 10 vector. And let's use relu at both steps. We know that this kinda works and converges quickly. In particular, after T steps we get error E(T) and E(1e6) 1024->1024->10, using the same relu? Same question, but now we replace relu with tanh: 2400->1024->10.
Re: Matrix Calculus for Deep Learning
#23A genuine question: is there any math behind ML at all? For example, is there any solid theory, with proven theorems, that would tell us what happens when we add another conv layer here or use a 3x3 conv kernel instead of a 2x2 one over there, or replace that tanh with a relu? From my limited understanding, ML roughly works like this: we shuffle around the ML graph, using some intuition, off-load it to a cluster of G…
I'd like to chime in that Variational Autoencoder's theoretical framework is also quite useful in unsupervised learning tasks. It bridges variational inference methods (a powerful classical ML technique) with neural networks, thus allow very efficient representation, generalization and disentanglement.
Re: Matrix Calculus for Deep Learning
#24A genuine question: is there any math behind ML at all? For example, is there any solid theory, with proven theorems, that would tell us what happens when we add another conv layer here or use a 3x3 conv kernel instead of a 2x2 one over there, or replace that tanh with a relu? From my limited understanding, ML roughly works like this: we shuffle around the ML graph, using some intuition, off-load it to a cluster of G…
There's plenty of math still. In order to not waste GPU-hours, you gotta have some intution as to what _not_ to do, because otherwise you'll be trying for a very long time and relying entirely on luck, which is combinatorially unfavorable and frustrating. That intuition is often grounded in math, mostly differential calculus and statistics. That said, there are a good number of techniques which nobody is sure why they work. I.e. batchnorm, which is used all the time. People have some theories and intuitions as to why it works, but nobody is sure why, nor is there any rigorous way to pick hyperparameters for it.
Re: Matrix Calculus for Deep Learning
#25Earlier quoted context omitted.
Of course there is. All the building blocks that people are mix and matching in networks nowadays were introduced at some point. The paper that introduced batch norm, adaptive instance norm, attention heads, or any module used in a network have an extensive discussion of the motivation for their existance, some derivation or proof that they do what you want, and an empirical test to show it helps in practice. The rea…
Actually, I read the batch norm paper and maybe I forgot important details, but it roughly went like this: "here we add a term `b` to make sure the mean values of Ax+b are zero and that will help us with convergence; ah, and here is a covariance matrix!", but no quantitative proofs about how much that convergence was helped. No, I intuitively agree that shifting the mean value to zero should help, but math taught me…
I think you and the person you're responding to might have slightly different expectations behind what level of rigor counts as "math", just like how physicists and theoretical mathematicians often have somewhat different ideas about rigor.
My impression is that obviously ML is guided by math and people want to have an understanding of why some things converge and others don't. But "in the field" many people just mess around with different set-ups and see what works (especially in deep learning). Maybe theory follows to explain why it worked. I think you're right that a lot of progress in the field is based on intuition and some reasoning (e.g. trying something like an inception network) more than derivations that show that a particular set-up should be successful. I get the impression that most low-level components are pretty well understood, but when they are stacked and combined it gets more complicated.
I would be very curious to see a video of your cat solving MNIST in 1 hour!
Re: Matrix Calculus for Deep Learning
#26A genuine question: is there any math behind ML at all? For example, is there any solid theory, with proven theorems, that would tell us what happens when we add another conv layer here or use a 3x3 conv kernel instead of a 2x2 one over there, or replace that tanh with a relu? From my limited understanding, ML roughly works like this: we shuffle around the ML graph, using some intuition, off-load it to a cluster of G…
There are maths and proves behind Neural Networks, but Machine Learning is an umbrella term. There are areas within it, maybe not that mainstream, that are also based on solid maths foundations.
One of the most interesting and elegant examples , Topological Data Analysis that is based on Topology and utilises tools like persistent homology. They can be applied to image processing, classification, etc.
Re: Matrix Calculus for Deep Learning
#27A genuine question: is there any math behind ML at all? For example, is there any solid theory, with proven theorems, that would tell us what happens when we add another conv layer here or use a 3x3 conv kernel instead of a 2x2 one over there, or replace that tanh with a relu? From my limited understanding, ML roughly works like this: we shuffle around the ML graph, using some intuition, off-load it to a cluster of G…
If I understand your question correctly, this is exactly what Bayes' Theorem deals with. Namely that B must be true within some probability given that B is true represented as P(A | B).
Let's consider a simple image classifier: 64x64 grayscale input, 10x1 output that detects 10 classes of images. The model: y=tanh(Ax+b). You'd probably say that there is no way this will work because this model is too simple. But can you explain why this won't work? Can you tell what the maximum accuracy this model can reach? What kind of datasets this model would work better on?
[Edit] Whether this model works is question. Sure, it can't recognize dogs or cats, but what if the dataset is 0-9 digits? Now it suddenly works, right? And works really well. But what's changed? Can we describe in mathematical terms what makes the 0-9 dataset so special? It'll probably work with A-J letters too, but what about hieroglyphs?
The math I'm looking for would tell that E(T) is the error and on such model and such dataset, E(T)=exp(-T^2)+O(exp(-T^3)), according to such and such theorem; and according to another theorem, if the dataset is isomorphic to that manifold, E(T) can be improved to O(exp(-3T^2)).
Re: Matrix Calculus for Deep Learning
#28If anybody want to read the real stuff then here is the reference: Matrix Computations https://www.cs.cornell.edu/cv/Books/GVL/index.htm
The book by Golub (RIP) and Van Loan goes way beyond what's discussed here (SVD, QR decomposition, eigenvalue computations, iterative solvers, error analysis, etc.), and it's focused more on (numeric) linear algebra rather than calculus.
Re: Matrix Calculus for Deep Learning
#29I’m in favor of there being more and better resources to learn anything out there, but every time I see a deep learning 101 type material all I can think is “who is this for?”. In ~July 2016 I was at a presentation by NVidia at GW in DC. They showed off how easy it was to build out and train a model using some of their tooling (Digits maybe?). After the demo they opened it up for questions and a grad student ‘asked’…
While this page looks nice, there is certainly a proliferation of beginner articles in this area. I think it's driven by demand, the same way gyms get tons of new members in January. As someone who sees all the hype and salaries involving ML, you dream of doing it too, so you look for articles to start out. That's the demand. After reading it, you don't get it and have no patience so you raise demand for titles like "ML for humans" or ML made easy or "Gentle intro to ML for the rest of us". Or ditch articles and watch Siraj ramble about making money with an AI startup today! I'd wager that only a small percentage of readers actually works their way through to advanced topics.
On the supply side (while TFA looks legit) people who are a few lessons ahead want to increase their visibility, start a blog/brand, make their CV stand out by showing community engagement and writing from a position of authority. This is mostly seen on Medium.
How to avoid the trap of being an eternal beginner? Accept that it will take time, be clear on your goals, try gathering a group of peers and expert guidance. Reddit and forums can be crap for this as you the beginner will gravitate towards the self proclaimed experts who may be full of shit and just play social games well, creating a blind leading the blind situation and cargo culting around terms that nobody really understands. There is a value in universities: they lay out a path, give guidance and let you work/learn together with peers. Ok, enough with this rant.
Re: Matrix Calculus for Deep Learning
#30A genuine question: is there any math behind ML at all? For example, is there any solid theory, with proven theorems, that would tell us what happens when we add another conv layer here or use a 3x3 conv kernel instead of a 2x2 one over there, or replace that tanh with a relu? From my limited understanding, ML roughly works like this: we shuffle around the ML graph, using some intuition, off-load it to a cluster of G…
Of course there is. All the building blocks that people are mix and matching in networks nowadays were introduced at some point. The paper that introduced batch norm, adaptive instance norm, attention heads, or any module used in a network have an extensive discussion of the motivation for their existance, some derivation or proof that they do what you want, and an empirical test to show it helps in practice. The rea…
nobody really knows how AI works is one of those myths told by the media
It's not a myth. No one really understands how neural networks work. We don't know why a particular model works well. Or why any model works well. For example no one can answer why NNs generalize so well even when they have enough learning capacity to memorize all training examples. We can guess, but we don't know for sure. Most of the proofs you see in papers are there as fillers, so that papers seem more convincing. We rarely can prove anything mathematically about NNs that has any practical value or leads to any breakthroughs in understanding.
If we did really understand how NNs work, then we wouldn't need to do expensive hyperparameter searches - we would have a way to determine the optimal ones given a particular architecture and training data. And we wouldn't need to do expensive architecture searches, yet the best of the latest convnets have been found through NAS (e.g. EfficientNet), and there's very little math involved in the process - it's pretty much just random search.
Funny you mentioned the batchnorm paper - we still don't know why batchnorm is so effective - the paper gave an explanation (covariate shift reduction) which later was shown to be wrong (batchnorm does not reduce it), then several other explanations were suggested (smoother loss surface, easier gradient flow, etc), but we still don't know for sure. Pretty much every good idea in NN field is a result of lots of experimentation, good intuition developed in the process, looking at how a brain does it, and practical constraints. And yes, sometimes we're looking at the equations, and thinking hard, and sometimes we see a better way to do stuff. But usually it starts with empirical tests, and if successful, some math is used in the attempt to explain things. Not the other way around.
NNs are currently at a similar point as where physics was before Newton and before calculus.