Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computational Linear Algebra

fast.ai

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Computational Linear Algebra

#31

Post by snarfy »

vog wrote: post #8
fdej wrote: post #7

> Locality: traditional runtime computations focus on Big O, the number of operations computed. However, for modern computing, moving data around in memory can be very time-consuming I need to nitpick here... Big O notation is a way to describe growth rates of functions. You can count data movements (or anything else) with Big O.

Moreover, "moving data around in memory can be very time-consuming" means that in the end, it is still about time, not about memory. So the correct way would be to translate memory access to time, but that means modelling the memory hierarchy, modelling caches in general, and finally perhaps modelling the specifically used caching strategies.

The limits on memory access are physical, illustrated by grass hoppers famous video about nanoseconds and the speed of light.

Should computer algorithms always assume they need to model caches since they are never going away? When determining the computational complexity, time and memory are treated with equivalence, but real memory doesn't and never will behave that way.

Re: Computational Linear Algebra

#32

Post by chris_st »

thearn4 wrote: post #29

Looks like a reasonable overview of applications of dense linear algebra operations, with very specific applications in mind. I feel like iterative Krylov subspace methods should be around somewhere, but to be honest I'm not sure what the applications space for linear inverse problems looks like in the deep learning domain. So maybe that wouldn't quite fit (despite me finding it to be pretty cool, and these methods s…

So, what applications are iterative Krylov subspace methods good for? Can you give us a sense of where they're useful, and what (in general) inverse methods in this space are?

Re: Computational Linear Algebra

#33

Post by jph00 »

thearn4 wrote: post #29

Looks like a reasonable overview of applications of dense linear algebra operations, with very specific applications in mind. I feel like iterative Krylov subspace methods should be around somewhere, but to be honest I'm not sure what the applications space for linear inverse problems looks like in the deep learning domain. So maybe that wouldn't quite fit (despite me finding it to be pretty cool, and these methods s…

Krylov subspace methods are heavily covered in the course. Trevathan is the main text we recommend in the course.

Re: Computational Linear Algebra

#34

Post by jph00 »

jph00 wrote: post #33
thearn4 wrote: post #29

Looks like a reasonable overview of applications of dense linear algebra operations, with very specific applications in mind. I feel like iterative Krylov subspace methods should be around somewhere, but to be honest I'm not sure what the applications space for linear inverse problems looks like in the deep learning domain. So maybe that wouldn't quite fit (despite me finding it to be pretty cool, and these methods s…

Krylov subspace methods are heavily covered in the course. Trevathan is the main text we recommend in the course.

Specifically, it's sections 9 and 10 here: https://github.com/fastai/numerical-linear-algebra/blob/mast...

Re: Computational Linear Algebra

#35

Post by jph00 »

wodenokoto wrote: post #27
lemming wrote: post #19

For someone with an ancient undergrad math background and only "interested observer" level of machine learning knowledge, would it be better to do this course before tackling the deep learning one?

No, start with fast.ai's deep learning course. They have a top down approach so you start with connecting layers to solve deep learning problems and then dig down into the theory behind the thing. If you want a bottom up approach, go look at Ng's coursera course. http://course.fast.ai

Absolutely right. There's very little linear algebra required in deep learning. This Computational Linear Algebra course mainly covers decompositions, which aren't used much in deep learning.

Re: Computational Linear Algebra

#36

Post by jph00 »

will_pseudonym wrote: post #24
lemming wrote: post #19

For someone with an ancient undergrad math background and only "interested observer" level of machine learning knowledge, would it be better to do this course before tackling the deep learning one?

If I wanted to learn linear algebra, I would reinvent PageRank.

It's a very helpful exercise, and something we do in lesson 9 of this course.

Re: Computational Linear Algebra

#37

Post by SAI_Peregrinus »

snarfy wrote: post #31
vog wrote: post #8

Earlier quoted context omitted.

Moreover, "moving data around in memory can be very time-consuming" means that in the end, it is still about time, not about memory. So the correct way would be to translate memory access to time, but that means modelling the memory hierarchy, modelling caches in general, and finally perhaps modelling the specifically used caching strategies.

The limits on memory access are physical, illustrated by grass hoppers famous video about nanoseconds and the speed of light. Should computer algorithms always assume they need to model caches since they are never going away? When determining the computational complexity, time and memory are treated with equivalence, but real memory doesn't and never will behave that way.

Grace Hopper, not grass hopper. The latter is an insect, the former was an awesome computer engineer who invented the compiler.

Re: Computational Linear Algebra

#38

Post by jph00 »

thearn4 wrote: post #30
rabreu08 wrote: post #2

Looks like a good course. I think it would benefit if they added some module on implementing some basic Linear system of equations solvers, like gradient or steepest descent. Or even GMRES/MINRES or so.. The amout of knowledge that i gained from trying to implement these was remarkable.

GMRES/Krylov spaces seem like they ought to have more of a place in deep learning. But maybe thats the mathematical part of me having a hammer and wanting to see everything as a nail. One possibility that I haven't seen: if you have a neural network graph with a small subgraph that is not 100% explicit (i.e. I have nodes with cyclic dependency that have to be solved collectively via newton-type method), the gradient…

Krylov spaces often come up in derivations for optimization methods in DL, and have been studied as an explicit technique on their own. Eg see https://arxiv.org/abs/1111.4259

However more recent theoretical and empirical results show that these kinds of approaches don't deal well with the huge saddle point problem in DL loss functions. But momentum based techniques work great.

Re: Computational Linear Algebra

#39

Post by knlje »

jph00 wrote: post #34
jph00 wrote: post #33

Earlier quoted context omitted.

Krylov subspace methods are heavily covered in the course. Trevathan is the main text we recommend in the course.

Specifically, it's sections 9 and 10 here: https://github.com/fastai/numerical-linear-algebra/blob/mast...

Just a remark. I cannot see sections 9 and 10 in your link. They only go up to section 8.

Re: Computational Linear Algebra

#40

Post by cdavid »

chris_st wrote: post #32
thearn4 wrote: post #29

Looks like a reasonable overview of applications of dense linear algebra operations, with very specific applications in mind. I feel like iterative Krylov subspace methods should be around somewhere, but to be honest I'm not sure what the applications space for linear inverse problems looks like in the deep learning domain. So maybe that wouldn't quite fit (despite me finding it to be pretty cool, and these methods s…

So, what applications are iterative Krylov subspace methods good for? Can you give us a sense of where they're useful, and what (in general) inverse methods in this space are?

(warning, I am not a numerical computation specialist)

Krylov subspaces are a (the ?) common framework to compute eigenvalues and singular values, and it works for sparse matrices as well. IOW, it is extremely useful for most problems that can be expressed as matrix factorization (e.g. recommendation using collaborative filtering, initial page rank-like algos).

Post reply on HN