Learning Math for Machine Learning • Hacker News Board

Re: Learning Math for Machine Learning

#101

Post by throwawaymath » Thu, Aug 02, 2018, 3:07 PM UTC

jacobolus wrote: post #99

throwawaymath wrote: post #47

Earlier quoted context omitted.

Speaking as one of the people who recommended it in this thread: I don't think math anxiety is the right focus for which textbook to choose. More precisely, I don't think you should try to solve that problem by getting a different linear algebra textbook. To put it bluntly, someone with math anxiety probably just doesn't have the mathematical maturity for linear algebra yet. In that case they'd be doing themselves a…

Tools from linear algebra can be accessible and useful to many people who don’t want to (or are not yet prepared to) prove nontrivial theorems. Indeed, a book like Axler’s should probably be used in a second semester-long linear algebra course for typical undergraduates wanting to study abstract mathematics; a gentler more concrete introduction would probably be better for students without previous exposure to linear…

> Those two audiences are very far apart.

Yes, that's precisely why I said, "This isn't Math 55: compared to Rudin and Halmos, Axler is a very accessible introduction to linear algebra for those who are ready for linear algebra."

How do you propose to teach linear algebra beyond basic matrix operations and Gaussian elimination if you're not teaching any theory? You can take some disparate tools from linear algebra (just like you can with analysis to make calculus), but The presentation of learning the mechanical tools of linear algebra versus the theory of linear algebra is a false dichotomy. Axler's textbook is a very nice compromise that provides students an understanding of why things are the way they are while still teaching them how to work through the numerical motions of things. You need not go so far as reading Finite Dimensional Vector Spaces if you want to avoid theory, but you need enough of it to put the mechanical operations in some kind of context.

Re: Learning Math for Machine Learning

#102

Post by throwawaymath » Thu, Aug 02, 2018, 3:09 PM UTC

pvg wrote: post #90

throwawaymath wrote: post #39

Earlier quoted context omitted.

> To play devil's advocate, probabilistic reasoning (probability theory, stochastic processes, Bayesian reasoning, graphical models, variational inference) might be equally if not more important. And of course, you're not going to get very far with probability theory and stochastic processes unless you have a mature understanding of analysis and measure theory :) This comment exchange neatly demonstrates the intrinsi…

to achieve that "comfortability" "comfort" is a perfectly cromulent word for this.

I was quoting the article; but thank you, I didn't know that. Good to know.

Re: Learning Math for Machine Learning

#103

Post by jacobolus » Thu, Aug 02, 2018, 3:17 PM UTC

throwawaymath wrote: post #101

jacobolus wrote: post #99

Earlier quoted context omitted.

Tools from linear algebra can be accessible and useful to many people who don’t want to (or are not yet prepared to) prove nontrivial theorems. Indeed, a book like Axler’s should probably be used in a second semester-long linear algebra course for typical undergraduates wanting to study abstract mathematics; a gentler more concrete introduction would probably be better for students without previous exposure to linear…

> Those two audiences are very far apart. Yes, that's precisely why I said, "This isn't Math 55: compared to Rudin and Halmos, Axler is a very accessible introduction to linear algebra for those who are ready for linear algebra." How do you propose to teach linear algebra beyond basic matrix operations and Gaussian elimination if you're not teaching any theory? You can take some disparate tools from linear algebra (j…

Personally I think that the undergraduate mathematics curriculum does a poor job of exposing people to examples and concrete situations before introducing new abstractions.

Students are often entirely unfamiliar with the context (problems, structures, goals, ...) for the new abstractions that are rained down on them, and end up treating their proofs as little exercises in symbol twiddling / pattern matching, without much understanding of what they are doing.

The undergraduate curriculum is put in this position because there is a lot of material to get through in not much time, and students are generally unprepared coming in. Ideally students would have a lot of exposure to basic material and lots of concrete examples starting in middle school or before, but that’s not where we are.

Re: Learning Math for Machine Learning

#104

Post by throwawaymath » Thu, Aug 02, 2018, 3:21 PM UTC

jacobolus wrote: post #103

throwawaymath wrote: post #101

Earlier quoted context omitted.

> Those two audiences are very far apart. Yes, that's precisely why I said, "This isn't Math 55: compared to Rudin and Halmos, Axler is a very accessible introduction to linear algebra for those who are ready for linear algebra." How do you propose to teach linear algebra beyond basic matrix operations and Gaussian elimination if you're not teaching any theory? You can take some disparate tools from linear algebra (j…

Personally I think that the undergraduate mathematics curriculum does a poor job of exposing people to examples and concrete situations before introducing new abstractions. Students are often entirely unfamiliar with the context (problems, structures, goals, ...) for the new abstractions that are rained down on them, and end up treating their proofs as little exercises in symbol twiddling / pattern matching, without…

I think we're in agreement on that point. In my experience most peoples' difficulty with higher mathematics comes from the tendency of elementary and high schools to push students along through grades without ensuring they've really mastered the material. Unfortunately most students come to hate math because they're introduced to ever more abstract and complex material when they haven't achieved a solid foundation to build upon. I don't see this artifact of our education system going away any time soon.

Re: Learning Math for Machine Learning

#105

Post by g9yuayon » Thu, Aug 02, 2018, 3:53 PM UTC

throwawaymath wrote: post #101

jacobolus wrote: post #99

Earlier quoted context omitted.

Tools from linear algebra can be accessible and useful to many people who don’t want to (or are not yet prepared to) prove nontrivial theorems. Indeed, a book like Axler’s should probably be used in a second semester-long linear algebra course for typical undergraduates wanting to study abstract mathematics; a gentler more concrete introduction would probably be better for students without previous exposure to linear…

> Those two audiences are very far apart. Yes, that's precisely why I said, "This isn't Math 55: compared to Rudin and Halmos, Axler is a very accessible introduction to linear algebra for those who are ready for linear algebra." How do you propose to teach linear algebra beyond basic matrix operations and Gaussian elimination if you're not teaching any theory? You can take some disparate tools from linear algebra (j…

Personally I used David Lay's Linear Algebra and Its Applications as my first linear algebra book. It's more formal than Gilbert Strang's famous text book, but less formal than Linear Algebra Done Right. It's emphasis on geometric intuition really struck home, particularly the discussion on coordinate systems, change of basis, and quadratic forms.

Re: Learning Math for Machine Learning

#106

Post by ivansavz » Thu, Aug 02, 2018, 4:59 PM UTC

jlelonm wrote: post #100

EnderMB wrote: post #91

Earlier quoted context omitted.

Seconded/Thirded. Hell, I have a CS degree, and my Maths knowledge is horrific. I didn't take Maths at A Level, so I stopped learning any Maths at 16. The course had Maths, but it's surprisingly easy to brute force a solution, and that was a good ten years ago now. My goal for years has been to learn enough Maths to be able to read Introduction to Algorithms and TAOCP without aid, and recently to be able to better un…

Fourthed(?) What's a linear transformation? I get that it's f(x + y) = f(x) + f(y) and f(cx) = c * f(x)... but what does that really mean? Why is the dot product equivalent to ||a||*||b|| cos C? I really have no idea, I just know the formulas.

> What's a linear transformation? > I get that it's f(x + y) = f(x) + f(y) and f(cx) = c * f(x)... > but what does that really mean?

That's a really good question!

Essentially, when some function (synonym for transformation) is linear, what this tells you is that it has "linear structure," which in turns out to be a very useful property to know about that function.

You can combine the two facts you mentioned above to obtain the equation

   f(a*x + b*y) = a*f(x) + b*f(y)

for which the interpretation is that the linear combination of inputs ax + by is transformed into the SAME linear combination of outputs af(x) + bf(y).

Suppose now that the input space of f can be characterized by some finite set of "directions" e.g. the x- and y-directions in case f is a 2D transformation, or perhaps the x-, y-, and z-directions if f is a 3D transformation. If f is a 3D transformation, using the linear property of f, it is possible to completely understand what f does by "probing" it with three "test inputs," one along each direction. Just input x, y, and z, and record the three output f(x), f(y), and f(z). Since you know f is linear, this probing with three vectors is enough to determine the output of f for any other input ax + by + cz --- the output will be af(x) + bf(y) + cf(z).

See the same explanations as above but in more details here: https://minireference.com/static/excerpts/noBSguide2LA_previ...

So why is this important? Well this "probing with a few input directions" turns out to be really useful. Basically, if f is non-linear, it means it super complicated and would be no simple way to describe what its outputs are for different inputs, but if it is linear then the "probing procedure" works. Furthermore, since both the inputs and outputs have the form of a linear combination (a constant times something + another constant times another thing + a third constant times a third thing), you can arrange these "things" into an array called a matrix and define an operation called "matrix multiplication" which performs the constant-times-something operation of outputs, when you give the constants as an array (vector) of inputs.

In summary, linear algebra is a bunch of machinery for expressing various transformations in terms of vectors and matrices in order to help with modelling various real-world phenomena ranging from computer graphics, biology, chemistry, graphs, crypto, etc. Even ML ;)

Re: Learning Math for Machine Learning

#107

Post by throwawaymath » Thu, Aug 02, 2018, 6:50 PM UTC

physicsguy wrote: post #92

throwawaymath wrote: post #36

Earlier quoted context omitted.

> The fact that so many software developers "don't like math" is ironic, because they're perfectly happy to manipulate symbols such as "x", "file", or "user_id" that stand in for other things every day. The entirety of mathematical knowledge is very much like a gigantic computer language (a formal system) in which every object is and must be precisely defined in terms of other objects, using and reusing symbols like…

> Meanwhile a professional programmer has to become good at things that mathematicians and scientists don’t have to care about, like version control or the idiosyncrasies of a specific language. Really depends on what kind of mathematician or scientist you are to be honest though. How good is someone's data analysis of an experiment if they can't reproduce it? Or if they've got 6 different versions of an application…

That's essentially my point. Programming and mathematics are so different from each other that, without special effort, a professional in one domain shouldn't be expected to be meaningfully better than average in the other.

This applies in both directions most mathematicians and scientists have such poor version control and development hygiene because mathematics doesn't imbue them with any special insight about how to be an engineer.

Re: Learning Math for Machine Learning

#108

Post by adw » Thu, Aug 02, 2018, 9:32 PM UTC

soVeryTired wrote: post #96

smallnamespace wrote: post #70

Earlier quoted context omitted.

For crystallography, the use of group theory in part originates in X-ray crystallography [1], where the goal is to take 2D projections of a repeating 3D structure (crystal), and use that along with other rules that you know to re-infer what the 3D structure is. Repeating structures have symmetries, so seeing the symmetries in your diffraction pattern inform you of the possible symmetries (and hence possible arrangeme…

I thought the work on the structure of DNA used Fourier analysis more than group theory. I know harmonic analysis in general combines the two, but I'm sure Crick and Watson could have done their work without knowing the definition of a group.

And by Crick and Watson you mean Crick, Watson, Franklin and Wilkins, right? It's fairly clear all four deserve at least partial authorship by modern standards. James Watson was a piece of work.

(https://www.theguardian.com/science/2015/jun/23/sexism-in-sc...)

Crick was absolutely certainly familiar with the crystallographic space groups; he was the student of Lawrence Bragg (https://en.wikipedia.org/wiki/Lawrence_Bragg), who is the youngest ever Nobel laureate in physics – winning it with his father for more or less inventing X-ray crystallography. It's mostly 19th-century mathematics, after all.

Re: Learning Math for Machine Learning

#109

Post by adw » Thu, Aug 02, 2018, 9:39 PM UTC

smallnamespace wrote: post #70

throwawaymath wrote: post #48

Earlier quoted context omitted.

Can you talk about the use of group theory for computer vision or crystallography a bit? I'm familiar with the math but I'm not familiar with group theory's applications in those areas. That sounds pretty interesting. Is it primarily group theory, or does it so venture into abstract algebra more generally?

For crystallography, the use of group theory in part originates in X-ray crystallography [1], where the goal is to take 2D projections of a repeating 3D structure (crystal), and use that along with other rules that you know to re-infer what the 3D structure is. Repeating structures have symmetries, so seeing the symmetries in your diffraction pattern inform you of the possible symmetries (and hence possible arrangeme…

> use that along with other rules that you know to re-infer what the 3D structure is

Great answer, thank you :-) Saved me a bunch of typing to explain it less well than you just did.

It's worth adding, for this crowd, that another way of thinking about the "other rules" you allude to is as a system of constraints; you can then set this up as an optimization problem (find the set of atomic positions minimizing reconstruction error under the set of symmetry constraints implied by the space group – so that means that solving crystal structures and machine learning are functionally isomorphic problems.

Re: Learning Math for Machine Learning

#110

Post by iamaaditya » Thu, Aug 02, 2018, 10:00 PM UTC

dsiegel2275 wrote: post #97

Does anyone have suggestions on learning resources for matrix calculus? I'm trying to come up to speed with the topic and could use pointers to worked examples, video lectures, etc.

[The Matrix Calculus You Need For Deep Learning](https://arxiv.org/pdf/1802.01528.pdf)