Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Machine Learn

docs.google.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: How to Machine Learn

#21

Post by j2kun »

mattbettinson wrote: post #2

I'm a freshman in university right now, how is linear algebra helpful in computer science? I'm finding it hard to stay motivated as I can't think of any uses outside of graphics. Maybe I'm just not far along in the course though.

I write a blog on math and programming and I see linear algebra applied every day.

1. Ranking in search engines (more generally, any kind of random walk analysis) [1]

2. Fourier analysis, and as a consequence most signal processing involves some understanding of linear algebra because integrals are linear. [2]

3. Regression [3] and more generally linear modeling of anything.

4. Facial recognition [4]

5. Community detection [5], where most leading methods analyze the spectrum of a graph to find communities. In fact, applied network science in general has a ton of linear algebra.

6. Greedy algorithms are characterized by a kind of generalization of linear systems [6]

7. Linear programming, perhaps the most applied piece of mathematics ever, needs a strong foundation of linear algebra [7]

8. All of quantum computing is literally just linear algebra [8].

9. Cryptography has a ton of linear algebra in it, and a large portion of the techniques are reasoned about with linear algebra.

10. Most of calculus relies on linear algebra, most importantly optimization [9]

11. Recent data analysis techniques based on topology do so through linear algebra [10]

12. Coding theory, including the algorithms used to correct errors on DVDs. Basically, any time you want to encode data so that you can recover from white noise, you're going to use a linear code. [11] This includes compression techniques.

13. Of course graphics.

I could go on...

[1]: http://jeremykun.com/2011/06/12/googles-pagerank-introductio...

[2]: http://jeremykun.com/2012/07/18/the-fast-fourier-transform/

[3]: http://jeremykun.com/2013/08/18/linear-regression/

[4]: http://jeremykun.com/2011/07/27/eigenfaces/

[5]: http://jeremykun.com/2014/05/19/community-detection-in-graph...

[6]: http://jeremykun.com/2014/08/26/when-greedy-algorithms-are-p...

[7]: http://jeremykun.com/2014/06/02/linear-programming-and-the-m...

[8]: http://jeremykun.com/2014/12/08/a-motivation-for-quantum-com...

[9]: http://jeremykun.com/2013/11/30/lagrangians-for-the-amnesiac...

[10]: http://jeremykun.com/2013/04/10/computing-homology/

[11]: http://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_corr...

Re: How to Machine Learn

#22

Post by dchapp »

fa wrote: post #20
dchapp wrote: post #16

Earlier quoted context omitted.

It's not that eigenvalues per se are useless--they're plainly not--but that no one finds eigenvalues in practice by computing the characteristic polynomial and solving for its roots. Unfortunately, that computation is often found in HW and exams in US undergraduate linear algebra courses.

This is especially tragic because matrix factorization algorithms are so deep and interesting, theory and programming-wise! LU, Cholesky, QR, eigendecomposition, SVD, mmmm. Round-off error tolerance, convergence criteria, stability, yum. Characteristic qualities and root finding: bleh.

Not to mention that when your matrix is 5x5 or more, there aren't even general solutions for roots if for some reason you're still insisting on the matrix->polynomial->eigenvalues route.

Re: How to Machine Learn

#23

Post by fa »

j2kun wrote: post #21
mattbettinson wrote: post #2

I'm a freshman in university right now, how is linear algebra helpful in computer science? I'm finding it hard to stay motivated as I can't think of any uses outside of graphics. Maybe I'm just not far along in the course though.

I write a blog on math and programming and I see linear algebra applied every day. 1. Ranking in search engines (more generally, any kind of random walk analysis) [1] 2. Fourier analysis, and as a consequence most signal processing involves some understanding of linear algebra because integrals are linear. [2] 3. Regression [3] and more generally linear modeling of anything. 4. Facial recognition [4] 5. Community det…

Have you done one on compressive sensing and the functional analysis roots of that field?

Re: How to Machine Learn

#24

Post by j2kun »

fa wrote: post #23
j2kun wrote: post #21

Earlier quoted context omitted.

I write a blog on math and programming and I see linear algebra applied every day. 1. Ranking in search engines (more generally, any kind of random walk analysis) [1] 2. Fourier analysis, and as a consequence most signal processing involves some understanding of linear algebra because integrals are linear. [2] 3. Regression [3] and more generally linear modeling of anything. 4. Facial recognition [4] 5. Community det…

Have you done one on compressive sensing and the functional analysis roots of that field?

Nope. There's never enough time :(

Re: How to Machine Learn

#25

Post by chas »

karmacondon wrote: post #9
fa wrote: post #6

Earlier quoted context omitted.

Well, if I remember my university's beginner linear algebra course, there were many topics on the syllabus only due to historic accident, ancestor worship, and theoretical necessities: I remember parallelepipeds, Cramer's rule, solving eigensystems by solving for a polynomial's zeros... Let me tell you how many times I've used parallelepipeds, Cramer's rule, or found eigenvalues via the quadratic formula in the 12 ye…

Seconded. Same for discrete mathematics and differential equations. Interesting to learn about, but pretty much worthless as soon as you set foot off of campus. I'd love to see comments from anyone who has practically used any of the information from those classes as a part of their daily duties as a programmer of any kind.

I've used differential equations building a physics-based optimization system for an industrial process. Symbolically solving parts of the system really increased its accuracy and stability.

Re: How to Machine Learn

#26

Post by Chinjut »

dchapp wrote: post #22
fa wrote: post #20

Earlier quoted context omitted.

This is especially tragic because matrix factorization algorithms are so deep and interesting, theory and programming-wise! LU, Cholesky, QR, eigendecomposition, SVD, mmmm. Round-off error tolerance, convergence criteria, stability, yum. Characteristic qualities and root finding: bleh.

Not to mention that when your matrix is 5x5 or more, there aren't even general solutions for roots if for some reason you're still insisting on the matrix->polynomial->eigenvalues route.

Sure there are (in many reasonable senses). Polynomial root extraction just isn't expressible in terms of addition, subtraction, multiplication, division, and nth roots alone. But that's ok; there's nothing magic about that particular set of operations, so as to make it the end-all, be-all.

Re: How to Machine Learn

#27

Post by east2west »

fa wrote: post #20
dchapp wrote: post #16

Earlier quoted context omitted.

It's not that eigenvalues per se are useless--they're plainly not--but that no one finds eigenvalues in practice by computing the characteristic polynomial and solving for its roots. Unfortunately, that computation is often found in HW and exams in US undergraduate linear algebra courses.

This is especially tragic because matrix factorization algorithms are so deep and interesting, theory and programming-wise! LU, Cholesky, QR, eigendecomposition, SVD, mmmm. Round-off error tolerance, convergence criteria, stability, yum. Characteristic qualities and root finding: bleh.

What is interesting is obviously personal, and I know you don't mean to denigrate some topics in general, but I want to caution people that characteristic equations and root finding are complex and far from useless. Numerical computation of eigen decomposition starts from finding the roots of the characteristic equations (in altered form), the eigen values, without which there is no computation of SVD. How the behaviors of roots change as coefficients vary are fundamental in control engineering. Newton's method holds up half of numerical optimization. Undergraduates don't have to learn them because others have worked out the details and implemented them in software.

By the way, Cramer's rule is useless for numerical computation, but it is immensely useful in theoretical work. It belongs to the vast body of work dealing with determinants before the rise of linear algebra. Determinant is the only obvious connection to algebra left in an undergraduate's linear algebra course, so I can understand people are turned off by it.

Re: How to Machine Learn

#28

Post by raz32dust »

Useful list. I would recommend adding one more resource for linear algebra/machine learning:

I absolutely enjoyed learning Linear Algebra from these beautiful lectures by Prof. Gilbert Strang (MIT): https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD9101...

Seriously, I gained a new found appreciation for Linear algebra after going through these lectures. You should go over some of these lectures even if you already know linear algebra - it might give you insights you never had before (it did, for me). Absolute must-watch if you are into machine learning or related areas.

Re: How to Machine Learn

#29

Post by RogerL »

karmacondon wrote: post #9
fa wrote: post #6

Earlier quoted context omitted.

Well, if I remember my university's beginner linear algebra course, there were many topics on the syllabus only due to historic accident, ancestor worship, and theoretical necessities: I remember parallelepipeds, Cramer's rule, solving eigensystems by solving for a polynomial's zeros... Let me tell you how many times I've used parallelepipeds, Cramer's rule, or found eigenvalues via the quadratic formula in the 12 ye…

Seconded. Same for discrete mathematics and differential equations. Interesting to learn about, but pretty much worthless as soon as you set foot off of campus. I'd love to see comments from anyone who has practically used any of the information from those classes as a part of their daily duties as a programmer of any kind.

I use it every single day. In the past I did all kinds of simulations for avionics and flight. I've used it for epidemiology studies, signal processing, augmented reality, machine vision for factories, and nowadays I'm using it in computer vision. I couldn't get anywhere without differential equations, linear algebra, and so on.

Discrete mathematics includes graph theory, discrete statistics, topology, OR, and so on.

College isn't meant to be votech. It's meant to expand your horizons. How would you go off and build a robot, write code for the NIH, work for a VR firm, write code for oil&gas exploration, program a drone if you didn't know this math? My only regret is that I didn't take more math.

I guess it is all taste, but I want the ability to just go and do what I want, and frankly, this sort of work is deeply interesting because it requires you to solve interesting problems. By that I mean that learning the API to Qt or Unity or something is not deeply interesting - it difficult to the extent that it is opaque/poorly documented. Once you learn the pattern to put something together in those frameworks the work becomes quite pedestrian (can you put a button here that ... yes, I can do that, yawn).

Post reply on HN