Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R, OpenMP, MKL, Disaster

jyotirmoy.net

Post reply on HN
Print view View original

21–30 of 43 posts

Re: R, OpenMP, MKL, Disaster

#21

Post by civilized »

keewee7 wrote: post #6

As a software developer forced to work with data scientists who refuse to learn Python there is nothing I hate more than R. R is good for explorative data analysis but useless for everything else.

Of course, if you read the article, you find out that the problem had nothing to do with R. It was a misconfiguration of the underlying linear algebra libraries that R (and Python and everything else) relies on. The author even made a minimal reproducible example in a single C script, no dependencies on R whatsoever.

I hear a lot of "R is bad, Python is Enterprise Production Quality (TM)" blather at my work. It's always because the people involved don't understand computers, don't read documentation, don't debug, don't do root cause analysis, and want to quickly pass off responsibility for their laziness and incompetence. Meanwhile I and my team are happily chugging away, producing millions of dollars of reliable value for my company in R year after year.

Python lags far behind R in wide swaths of data science. Pandas is inferior to both dplyr and data.table, and R's modeling capabilities blow Python's out of the water in breadth and depth. You only use Python when you have to, e.g. for unstructured data and deep learning type stuff.

If your colleagues make you deal with their bad R code, that's too bad, but don't blame the language. It's designed to be easy to use, so a lot of bad coders use it. Go train your bad coders or hire better ones.

Re: R, OpenMP, MKL, Disaster

#22

Post by gnufx »

It's perhaps worth saying that if you must mix OpenMP libraries built against the LLVM (Kuck/Intel) runtime, and GNU GOMP on GNU/Linux: Ensure libomp is built with GOMP compatibility, however that's configured, make a shim from the result, like

    gcc -shared -Wl,-soname=libgomp.so.1 -o libgomp.so.1 empty.c -lomp5
where empty.c is an empty file, and put the result on LD_LIBRARY_PATH ahead of the real libgomp. Alternatively, preload the compatible libomp5. On Debian 11 there's already a libgomp in the llvm packaging. Dynamic linking assumed, as is right and fitting.

Re: R, OpenMP, MKL, Disaster

#23

Post by mellavora »

keewee7 wrote: post #6

As a software developer forced to work with data scientists who refuse to learn Python there is nothing I hate more than R. R is good for explorative data analysis but useless for everything else.

with regard to good software engineering, there is a funny thing about python. Simply cut-and-paste code from one env to another can completely destroy the program, if the cut-and-paste messes with the indentation.

Now some people say this can be solved with a good IDE. Which might (or might not) be true if you can reliably identify, by manually reviewing the code, the ends of the functions, loops, etc which got munged in the paste.

But interestingly enough, jupyter notebooks (which seem to be the go-to tool these days) aren't IDEs. Making it incredibly easy to fubar otherwise perfectly working code by pasting it from your local IDE into, let's say an AWS Sagemaker instance, to pick one random example of a current widely used jupyter implementation. So even if the problem could be fixed by a good IDE, there is no guarantee that that IDE is (easily) accessible for production code.

I just have a hard time seeing how such a fundamental flaw in a language can lead to "good software engineering"

Re: R, OpenMP, MKL, Disaster

#24

Post by vhhn »

microtonal wrote: post #5
kettleballroll wrote: post #3

In a previous life, almost a decade ago, I fought very similar fights with OpenMP and MKL using R. It's painful and you need to pay heed to all these small details pointed out in the docs as in OPs case. However, it's worth noting that OpenBLAS is as fast as MKL, at least if you compile it yourself for your system (i would expect that system provided ones with system detection would be as good, but that wasn't always…

OpenBLAS OpenBLAS is incompatible with application threads. Most Linux distributions provide a multi-threaded OpenBLAS that burns in a fire if you use it in multi-threaded applications. Even though OpenBLAS' performance is great, I'd be careful to give a general recommendation for people to rely on OpenBLAS. Like this MKL example, you have to be aware of its threading issues, read the documentation and compile it wit…

> OpenBLAS is incompatible with application threads

R is single-threaded.

Re: R, OpenMP, MKL, Disaster

#25

Post by laichzeit0 »

civilized wrote: post #21
keewee7 wrote: post #6

As a software developer forced to work with data scientists who refuse to learn Python there is nothing I hate more than R. R is good for explorative data analysis but useless for everything else.

Of course, if you read the article, you find out that the problem had nothing to do with R. It was a misconfiguration of the underlying linear algebra libraries that R (and Python and everything else) relies on. The author even made a minimal reproducible example in a single C script, no dependencies on R whatsoever. I hear a lot of "R is bad, Python is Enterprise Production Quality (TM)" blather at my work. It's alw…

I would completely concede that R has better libraries. However, getting stuff like online prediction into production is a real pain when the models are developed in R. And R is single threaded. There is no way to hide that detail.

Re: R, OpenMP, MKL, Disaster

#26

Post by civilized »

laichzeit0 wrote: post #25
civilized wrote: post #21

Earlier quoted context omitted.

Of course, if you read the article, you find out that the problem had nothing to do with R. It was a misconfiguration of the underlying linear algebra libraries that R (and Python and everything else) relies on. The author even made a minimal reproducible example in a single C script, no dependencies on R whatsoever. I hear a lot of "R is bad, Python is Enterprise Production Quality (TM)" blather at my work. It's alw…

I would completely concede that R has better libraries. However, getting stuff like online prediction into production is a real pain when the models are developed in R. And R is single threaded. There is no way to hide that detail.

R isn't the best for production predictions for sure (it can work though). But it's not hard to translate well-designed R processing pipelines and models into other languages if you must. The problem is that R programmers often don't know how to write good code in any language.

Same issue as Excel, really. Easy to use, so you get a lot of users with very thin engineering skills.

The solution is for production engineers to understand just enough R to set standards for data scientist code that enable reliable translation of the models to the production language. As with JS, you can complain about the yucky parts, or you can accept that it's the best tool for some jobs and make an effort to work around the yucky parts, or use the tools of those who are doing that (e.g. tidyverse and Wickham).

If you want data scientists to produce production-ready results, you have to hold them to the standards of production engineering.

Re: R, OpenMP, MKL, Disaster

#27

Post by deng »

laichzeit0 wrote: post #25
civilized wrote: post #21

Earlier quoted context omitted.

Of course, if you read the article, you find out that the problem had nothing to do with R. It was a misconfiguration of the underlying linear algebra libraries that R (and Python and everything else) relies on. The author even made a minimal reproducible example in a single C script, no dependencies on R whatsoever. I hear a lot of "R is bad, Python is Enterprise Production Quality (TM)" blather at my work. It's alw…

I would completely concede that R has better libraries. However, getting stuff like online prediction into production is a real pain when the models are developed in R. And R is single threaded. There is no way to hide that detail.

> And R is single threaded. There is no way to hide that detail.

Python isn't much better in this regard, thanks to the GIL.

What I actually found most baffling when I delved into R is the fact that it doesn't support 64bit integers (lack of proper native UTF-8 support coming a close second).

Re: R, OpenMP, MKL, Disaster

#28

Post by mellavora »

civilized wrote: post #26
laichzeit0 wrote: post #25

Earlier quoted context omitted.

I would completely concede that R has better libraries. However, getting stuff like online prediction into production is a real pain when the models are developed in R. And R is single threaded. There is no way to hide that detail.

R isn't the best for production predictions for sure (it can work though). But it's not hard to translate well-designed R processing pipelines and models into other languages if you must. The problem is that R programmers often don't know how to write good code in any language. Same issue as Excel, really. Easy to use, so you get a lot of users with very thin engineering skills. The solution is for production enginee…

"Same issue as Excel, really. Easy to use, so you get a lot of users with very thin engineering skills."

Huh?

While I totally agree with your quote, I'd think it applied a lot more to python than to R. Especially given that python seems to be the dominant "first language for people to learn when they get into programming" because it is "easy".

Re: R, OpenMP, MKL, Disaster

#29

Post by civilized »

mellavora wrote: post #28
civilized wrote: post #26

Earlier quoted context omitted.

R isn't the best for production predictions for sure (it can work though). But it's not hard to translate well-designed R processing pipelines and models into other languages if you must. The problem is that R programmers often don't know how to write good code in any language. Same issue as Excel, really. Easy to use, so you get a lot of users with very thin engineering skills. The solution is for production enginee…

"Same issue as Excel, really. Easy to use, so you get a lot of users with very thin engineering skills." Huh? While I totally agree with your quote, I'd think it applied a lot more to python than to R. Especially given that python seems to be the dominant "first language for people to learn when they get into programming" because it is "easy".

The proportion in R is higher because the community of software engineers working in R is a lot smaller. R coders are overwhelmingly data analysts, while Python coders have more diverse roles. People who use R are also much more likely to have learned R, and only R, from their university courses towards a data science-related degree, especially if that degree is in statistics.

Re: R, OpenMP, MKL, Disaster

#30

Post by boulos »

I don't have MKL to try this out, but I'd check that the MKL threading choice actually didn't break the initialization to 1.0 loop.

That is, instead of checking after doing the x[i] *= SCALE bit with cblas, I would check both before and after the scaling.

Post reply on HN