Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Computational Linear Algebra

fast.ai

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Computational Linear Algebra

#51

Post by zzazzdsa »

SAI_Peregrinus wrote: post #37
snarfy wrote: post #31

Earlier quoted context omitted.

The limits on memory access are physical, illustrated by grass hoppers famous video about nanoseconds and the speed of light. Should computer algorithms always assume they need to model caches since they are never going away? When determining the computational complexity, time and memory are treated with equivalence, but real memory doesn't and never will behave that way.

Grace Hopper, not grass hopper. The latter is an insect, the former was an awesome computer engineer who invented the compiler.

Then again, both have something to do with bugs on some level...

Re: Computational Linear Algebra

#52

Post by edshiro »

chris_st wrote: post #32
thearn4 wrote: post #29

Looks like a reasonable overview of applications of dense linear algebra operations, with very specific applications in mind. I feel like iterative Krylov subspace methods should be around somewhere, but to be honest I'm not sure what the applications space for linear inverse problems looks like in the deep learning domain. So maybe that wouldn't quite fit (despite me finding it to be pretty cool, and these methods s…

So, what applications are iterative Krylov subspace methods good for? Can you give us a sense of where they're useful, and what (in general) inverse methods in this space are?

I remember using Conjugate Gradient (https://en.wikipedia.org/wiki/Conjugate_gradient_method) for my 4th year university project, which involved finding the steady state of large and complex systems (e.g. railway signalling systems), that were expressed as sparse matrices with millions of rows/columns. It worked really well and executed faster than other iterative methods such as Jacoby/Gauss-Seidel, but the conjugate gradient method was a bit unstable and at times I would not be able to converge toward a solution - I think it was due to the high number of multiplications involved which may cause over-/underflow (but I can't remember).

The matrices themselves are a representation of a Markov chain.

Re: Computational Linear Algebra

#53

Post by snarfy »

SAI_Peregrinus wrote: post #37
snarfy wrote: post #31

Earlier quoted context omitted.

The limits on memory access are physical, illustrated by grass hoppers famous video about nanoseconds and the speed of light. Should computer algorithms always assume they need to model caches since they are never going away? When determining the computational complexity, time and memory are treated with equivalence, but real memory doesn't and never will behave that way.

Grace Hopper, not grass hopper. The latter is an insect, the former was an awesome computer engineer who invented the compiler.

Well, crap. It was a 4am post, and too late to edit now.

Re: Computational Linear Algebra

#54

Post by flor1s »

Thanks for sharing this, it seems like a lot of interesting material is being discussed. The audience seems to be more like the hacker news visitor than the average student though, as it feels like little hand holding is provided.

I've just started lecture 1 but I already felt some minor frustrations:

- One of the links in the first lecture is to a notebook about intro to convolutions but that notebook is just a big code dump.

- After executing the exercises, you lose the expected answer. It might be better if the answers were included as a comment in the code fragment.

- Sometimes the given answers are not actually the answer but just the computation performed as part of getting the answer. I.e. for the matrix-matrix products section in lecture 1 the suggested answer is just the resulting matrix from doing the matrix product, but according to the question in the text the answer should be the actual cheapest shop.

- Is this a USF course or a fast.ai course?

I don't know if the author is planning on improving the material, because right now it feels a bit like a beta version.

Re: Computational Linear Algebra

#55

Post by jph00 »

kxyvr wrote: post #50
jph00 wrote: post #49

Earlier quoted context omitted.

Exactly. Saddle points are like a magnet for them!

Absolutely not. Krylov methods are one method for solving a linear system and can absolutely be used with stochastic optimization methods. Generally speaking, most optimization algorithms require the solution of some linear system and we can use a Krylov method or we can use a direct solve or we can integrate the direct solve with the Krylov method as a preconditioner. However, this has nothing to do with the stochas…

Whilst what you say is accurate, it has nothing to do with my comment which you are replying to, which has specifically about the shape of the loss functions in deep learning, and recent research on how it impacts optimization of these functions.

Re: Computational Linear Algebra

#56

Post by kxyvr »

jph00 wrote: post #55
kxyvr wrote: post #50

Earlier quoted context omitted.

Absolutely not. Krylov methods are one method for solving a linear system and can absolutely be used with stochastic optimization methods. Generally speaking, most optimization algorithms require the solution of some linear system and we can use a Krylov method or we can use a direct solve or we can integrate the direct solve with the Krylov method as a preconditioner. However, this has nothing to do with the stochas…

Whilst what you say is accurate, it has nothing to do with my comment which you are replying to, which has specifically about the shape of the loss functions in deep learning, and recent research on how it impacts optimization of these functions.

The grand parent comment asked, "In short, are the Krylov methods (or perhaps other approaches) a bit too precise in their attraction to critical points?" to which you replied, "Exactly. Saddle points are like a magnet for them!" There's little room for ambiguity here and this is categorically false. Your linear solver has nothing to do with saddle points and Krylov methods do not attract them.

Re: Computational Linear Algebra

#57

Post by jjtheblunt »

thearn4 wrote: post #29

Looks like a reasonable overview of applications of dense linear algebra operations, with very specific applications in mind. I feel like iterative Krylov subspace methods should be around somewhere, but to be honest I'm not sure what the applications space for linear inverse problems looks like in the deep learning domain. So maybe that wouldn't quite fit (despite me finding it to be pretty cool, and these methods s…

That book is superlative! We taught from it at Urbana-Champaign even 20 years ago in grad school..it's still fantastic.
Post reply on HN