R, OpenMP, MKL, Disaster • Hacker News Board

Re: R, OpenMP, MKL, Disaster

#11

Post by WhompingWindows » Sun, Oct 03, 2021, 12:43 PM UTC

keewee7 wrote: post #6

As a software developer forced to work with data scientists who refuse to learn Python there is nothing I hate more than R. R is good for explorative data analysis but useless for everything else.

Wow, tell us how you really feel. How much have you used R and Python? Maybe those data scientists would prefer if you didn't viscerally hate the main data/statistics language and didn't call it useless for things beyond a narrow use-case. It may lead to better outcomes if people hated things less and tried to understand the valid use-cases, for instance the reams and reams of statistics that can be done on R where Python may lag behind, since R is the lingua franca of statistics and research.

Re: R, OpenMP, MKL, Disaster

#12

Post by microtonal » Sun, Oct 03, 2021, 12:45 PM UTC

jabl wrote: post #8

microtonal wrote: post #5

Earlier quoted context omitted.

OpenBLAS OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Even though OpenBLAS' performance is great, I'd be careful to give a general recommendation for people to rely on OpenBLAS. Like this MKL example, you have to be aware of its threading issues, read the documentation and compile it wit…

> OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Can you explain what you mean by this? Are you saying there's a correctness issue here? I only recall running into issues with MPI, where you (typically) run one MPI rank (process) per CPU core. Then if you combine that with a multi-threaded…

Can you explain what you mean by this?

There is a nice description of this:

https://github.com/xianyi/OpenBLAS/issues/2543

At a previous employer, we have seen various issues, including crashes, non-determinisms, etc. Usually, these issues would go away when switching to MKL.

Re: R, OpenMP, MKL, Disaster

#13

Post by otabdeveloper4 » Sun, Oct 03, 2021, 12:57 PM UTC

keewee7 wrote: post #6

As a software developer forced to work with data scientists who refuse to learn Python there is nothing I hate more than R. R is good for explorative data analysis but useless for everything else.

R is way more powerful and flexible for data science stuff. (Going from Python to R is almost like going from Excel to Python.)

Re: R, OpenMP, MKL, Disaster

#14

Post by kergonath » Sun, Oct 03, 2021, 1:00 PM UTC

microtonal wrote: post #5

kettleballroll wrote: post #3

In a previous life, almost a decade ago, I fought very similar fights with OpenMP and MKL using R. It's painful and you need to pay heed to all these small details pointed out in the docs as in OPs case. However, it's worth noting that OpenBLAS is as fast as MKL, at least if you compile it yourself for your system (i would expect that system provided ones with system detection would be as good, but that wasn't always…

OpenBLAS OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Even though OpenBLAS' performance is great, I'd be careful to give a general recommendation for people to rely on OpenBLAS. Like this MKL example, you have to be aware of its threading issues, read the documentation and compile it wit…

> OpenBLAS is incompatible with application threads.

I’ve never had any issue when using it in OpenMP codes (either compiling it myself or using the libopenblas_omp.so present in some distros), what do you mean by “burn in a fire”?

Re: R, OpenMP, MKL, Disaster

#15

Post by gjvc » Sun, Oct 03, 2021, 1:03 PM UTC

microtonal wrote: post #5

kettleballroll wrote: post #3

In a previous life, almost a decade ago, I fought very similar fights with OpenMP and MKL using R. It's painful and you need to pay heed to all these small details pointed out in the docs as in OPs case. However, it's worth noting that OpenBLAS is as fast as MKL, at least if you compile it yourself for your system (i would expect that system provided ones with system detection would be as good, but that wasn't always…

OpenBLAS OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Even though OpenBLAS' performance is great, I'd be careful to give a general recommendation for people to rely on OpenBLAS. Like this MKL example, you have to be aware of its threading issues, read the documentation and compile it wit…

> The BLAS/LAPACK ecosystem is a mess. I wish that Intel would just open source MKL and properly support AMD CPUs.

Given that their latest compilers are based on LLVM, that seems like a fair trade between the closed- and open-source worlds.

Re: R, OpenMP, MKL, Disaster

#16

Post by kzrdude » Sun, Oct 03, 2021, 1:06 PM UTC

Systems that fail without producing an error or warning are worrying and unsettling to me.

Re: R, OpenMP, MKL, Disaster

#17

Post by gnufx » Sun, Oct 03, 2021, 1:12 PM UTC

microtonal wrote: post #5

kettleballroll wrote: post #3

In a previous life, almost a decade ago, I fought very similar fights with OpenMP and MKL using R. It's painful and you need to pay heed to all these small details pointed out in the docs as in OPs case. However, it's worth noting that OpenBLAS is as fast as MKL, at least if you compile it yourself for your system (i would expect that system provided ones with system detection would be as good, but that wasn't always…

OpenBLAS OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Even though OpenBLAS' performance is great, I'd be careful to give a general recommendation for people to rely on OpenBLAS. Like this MKL example, you have to be aware of its threading issues, read the documentation and compile it wit…

Debian and Fedora provide serial, OpenMP, and pthreads versions of lilbopenblas. Are you sure OpenBLAS doesn't detect nested OpenMP? I thought it did, though I'd normally use the serial version outside something like R, but if you mix different low-level simple pthreads with high-level OpenMP, you can expect problems. OpenBLAS is fine generally -- competitive with MKL on Intel hardware and infinitely faster on ARM and POWER. For PyTorch, presumably you want libxsmm (which is responsible for MKL's current small matrix performance). On AMD hardware, I don't understand why people avoid AMD's support, which is just a version of BLIS and libflame. (BLIS' OpenMP story seems better than OpenBLAS'.) The linear algebra story on GNU/Linux distributions would be less of a mess without proprietary libraries like MKL. It's fine if you take the Debian approach, in significant experience running heterogeneous HPC systems. Fedora has cocked up policy through not listening to such experience, but you can do the Debian-style thing with the approach of https://loveshack.fedorapeople.org/blas-subversion.html (and see the old R example refuting the MKL story). That's one example of the value of dynamic linking.

Re: R, OpenMP, MKL, Disaster

#18

Post by gnufx » Sun, Oct 03, 2021, 1:19 PM UTC

keewee7 wrote: post #6

As a software developer forced to work with data scientists who refuse to learn Python there is nothing I hate more than R. R is good for explorative data analysis but useless for everything else.

I've never seen anything for Python that allows you to a linear algebra-based code and run it at maybe petascale with trivial modifications. There's an R example somewhere under https://pbdr.org/publications.html

Re: R, OpenMP, MKL, Disaster

#19

Post by microtonal » Sun, Oct 03, 2021, 1:24 PM UTC

gnufx wrote: post #17

microtonal wrote: post #5

Earlier quoted context omitted.

OpenBLAS OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Even though OpenBLAS' performance is great, I'd be careful to give a general recommendation for people to rely on OpenBLAS. Like this MKL example, you have to be aware of its threading issues, read the documentation and compile it wit…

Debian and Fedora provide serial, OpenMP, and pthreads versions of lilbopenblas. Are you sure OpenBLAS doesn't detect nested OpenMP? I thought it did, though I'd normally use the serial version outside something like R, but if you mix different low-level simple pthreads with high-level OpenMP, you can expect problems. OpenBLAS is fine generally -- competitive with MKL on Intel hardware and infinitely faster on ARM an…

On AMD hardware, I don't understand why people avoid AMD's support, which is just a version of BLIS and libflame.

A year ago, I benchmarked a transformer network with libtorch linked against various BLAS libraries (numbers are in sentences per second, MKL with CPU detection override on AMD, 4 threads):

Ryzen 3700X - OpenBLAS: 83, BLIS: 69, AMD BLIS: 80, MKL: 119

Xeon Gold 6138 - OpenBLAS: 88, BLIS: 52, AMD BLIS: 59, MKL: 128

I guess people avoid AMD's support, because MKL is just much faster? AMD BLIS did add batch GEMM support since then. Didn't have time to try that out yet.

Re: R, OpenMP, MKL, Disaster

#20

Post by syockit » Sun, Oct 03, 2021, 1:32 PM UTC

microtonal wrote: post #12

jabl wrote: post #8

Earlier quoted context omitted.

> OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Can you explain what you mean by this? Are you saying there's a correctness issue here? I only recall running into issues with MPI, where you (typically) run one MPI rank (process) per CPU core. Then if you combine that with a multi-threaded…

Can you explain what you mean by this? There is a nice description of this: https://github.com/xianyi/OpenBLAS/issues/2543 At a previous employer, we have seen various issues, including crashes, non-determinisms, etc. Usually, these issues would go away when switching to MKL.

One of the more painful issues is hanging (lockup) at full CPU usage. At my workplace, initially we introduced a timeout to workaround the hang while trying to determine the cause of the hang. It happened within multithread R code. Various build flags for OpenBLAS have been tried to no avail. Setting OPENBLAS_NUM_THREADS=1 surely makes the problem go away, at the expense of performance.

That R code has since been ported to Python, but we faced the same issue again when using ThreadPoolExecutor, so we had to change it into ProcessPoolExecutor instead.