Eigen: A C++ template library for linear algebra • Hacker News Board

Re: Eigen: A C++ template library for linear algebra

#41

Post by waynecochran » Mon, Apr 04, 2022, 5:18 PM UTC

Probably won't ever catch Fortran, but using Eigen templates for reductions really opens the door for compile time optimizations; e.g. these are all reductions that do the same thing

    const float residual = (L.array() * P.array()).colwise().sum().square().mean();
    const float residual = L.cwiseProduct(P).array().colwise().sum().array().square().mean();
    const float residual = (L.transpose() * P).diagonal().array().square().mean();

The compiler can optimize using static information, e.g. these would all be handled differently for the following types

    Eigen::Matrix L(3,3), P(3,3);
    Eigen::Matrix L(3,K), P(3,K);
    Eigen::Matrix L(N,K), P(N,K);

Re: Eigen: A C++ template library for linear algebra

#42

Post by markisus » Mon, Apr 04, 2022, 5:19 PM UTC

Eigen is really nice for getting code that looks like the underlying math and it also optimizes away unnecessary temporaries by using template expressions. However, when you mess up, you get a screen full of template errors that take some experience to understand and debug.

It makes me wish for a language + compiler where linear algebraic objects are first-class values.

Re: Eigen: A C++ template library for linear algebra

#43

Post by joelhaasnoot » Mon, Apr 04, 2022, 5:20 PM UTC

Ahh, the horror of getting this to compile last week in Python :(

Re: Eigen: A C++ template library for linear algebra

#44

Post by tspooner » Mon, Apr 04, 2022, 5:27 PM UTC

markisus wrote: post #42

Eigen is really nice for getting code that looks like the underlying math and it also optimizes away unnecessary temporaries by using template expressions. However, when you mess up, you get a screen full of template errors that take some experience to understand and debug. It makes me wish for a language + compiler where linear algebraic objects are first-class values.

While it's not exactly what you're asking for, I do find the LinearAlgebra (standard) library in Julia to be pretty fantastic: https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/

Re: Eigen: A C++ template library for linear algebra

#45

Post by Symmetry » Mon, Apr 04, 2022, 5:29 PM UTC

waynecochran wrote: post #41

Probably won't ever catch Fortran, but using Eigen templates for reductions really opens the door for compile time optimizations; e.g. these are all reductions that do the same thing const float residual = (L.array() * P.array()).colwise().sum().square().mean(); const float residual = L.cwiseProduct(P).array().colwise().sum().array().square().mean(); const float residual = (L.transpose() * P).diagonal().array().squar…

The compile time loop fusion is also particularly nice.

Re: Eigen: A C++ template library for linear algebra

#46

Post by Sesse__ » Mon, Apr 04, 2022, 5:34 PM UTC

jhartmann wrote: post #29

phkahler wrote: post #11

My only complaint is that using OpenMP Eigen can be slower with SMT than without SMT. They even suggest telling to use half as many threads as you have "cores" when "cores" means twice as many due to SMT. Otherwise, we've seen a 8-10x performance increase in SolveSpace (CAD) in some situations after switching from home-grown matrix operations to Eigen.

This has been true for years, Intel CPU's can't efficiently perform math operations when hyperthreading is involved. Generally there is only a single shared FPU/AVX/SSE unit doing the math over two hyperthreads. Since the Eigen implementation often can keep that unit 100% busy, it makes no sense to try and run two threads at full tilt through the units. I tested all this very heavily before Eigen had AVX-512 support.…

> Generally there is only a single shared FPU/AVX/SSE unit doing the math over two hyperthreads.

Hyperthreading does in general share units (both ALU and others); that's what hyperthreading is.

Apart from that, it really depends on what operations you're doing; e.g., modern Intel CPUs have three ports that can issue a 256-bit FMA, each, every cycle.

Re: Eigen: A C++ template library for linear algebra

#47

Post by bee_rider » Mon, Apr 04, 2022, 5:35 PM UTC

Const-me wrote: post #40

bee_rider wrote: post #37

Earlier quoted context omitted.

That's interesting. I'd assume that a template library like Eigen would be most competitive for really small matrices and vectors of a known size -- it should have a fundamental advantage over something like MKL or BLIS, in that it can fully inline and unroll everything if it wants, right?

My guess it’s their deliberate design decision. It could be that most users of the library don’t care about small things. About the template stuff, I think their main performance advantage over traditional BLAS is not even SIMD, it’s lazy evaluation. Expressions like x=a*b+c never compute the complete a*b matrix or vector. The a*b expression returns a small placeholder object on the stack, of a scary type with couple…

Yeah, lazy evaluation/fusing operations is I think their best feature. Fixed interface BLAS has really good performance, but I mean, something as simple as DAXPBY is an extension. Selecting what functionality to super-optimize is hard for a fixed interface, and involves non-technical stuff like figuring out what subroutines people actually want.

I'm sure someone has already looked at this, but I wonder if Eigen can just somehow nab kernels directly from BLIS, haha.

Re: Eigen: A C++ template library for linear algebra

#48

Post by liquidify » Mon, Apr 04, 2022, 5:37 PM UTC

Const-me wrote: post #40

bee_rider wrote: post #37

Earlier quoted context omitted.

That's interesting. I'd assume that a template library like Eigen would be most competitive for really small matrices and vectors of a known size -- it should have a fundamental advantage over something like MKL or BLIS, in that it can fully inline and unroll everything if it wants, right?

My guess it’s their deliberate design decision. It could be that most users of the library don’t care about small things. About the template stuff, I think their main performance advantage over traditional BLAS is not even SIMD, it’s lazy evaluation. Expressions like x=a*b+c never compute the complete a*b matrix or vector. The a*b expression returns a small placeholder object on the stack, of a scary type with couple…

This is called "expression templates" https://en.wikipedia.org/wiki/Expression_templates

Re: Eigen: A C++ template library for linear algebra

#49

Post by phkahler » Mon, Apr 04, 2022, 5:40 PM UTC

Sesse__ wrote: post #46

jhartmann wrote: post #29

Earlier quoted context omitted.

This has been true for years, Intel CPU's can't efficiently perform math operations when hyperthreading is involved. Generally there is only a single shared FPU/AVX/SSE unit doing the math over two hyperthreads. Since the Eigen implementation often can keep that unit 100% busy, it makes no sense to try and run two threads at full tilt through the units. I tested all this very heavily before Eigen had AVX-512 support.…

> Generally there is only a single shared FPU/AVX/SSE unit doing the math over two hyperthreads. Hyperthreading does in general share units (both ALU and others); that's what hyperthreading is. Apart from that, it really depends on what operations you're doing; e.g., modern Intel CPUs have three ports that can issue a 256-bit FMA, each, every cycle.

>> Hyperthreading does in general share units (both ALU and others); that's what hyperthreading is.

Yes, I think the issue with Eigen is cache related. They apparently have optimizations that are aware of cache architecture and running 2 threads that share the same cache will screw that up, resulting in more misses. If this is the case, I'd prefer algorithms that are cache line size agnostic. It is still much faster than the simple hand-written code we had before!

Re: Eigen: A C++ template library for linear algebra

#50

Post by deng » Mon, Apr 04, 2022, 5:47 PM UTC

markisus wrote: post #42

Eigen is really nice for getting code that looks like the underlying math and it also optimizes away unnecessary temporaries by using template expressions. However, when you mess up, you get a screen full of template errors that take some experience to understand and debug. It makes me wish for a language + compiler where linear algebraic objects are first-class values.

> It makes me wish for a language + compiler where linear algebraic objects are first-class values.

Like Fortran?