Python, Machine Learning, and Language Wars (2015)

I did that earlier today:

pip install jupyter

pip install numpy

pip install scipy

pip install scikit-learn

pip install matplotlib

The only problem I had was with OpenCV, which requires manual make installation if you want the contrib package. The other problem was when trying to install scikit-learn, it requires manual pip installation of scipy.

People have been upset about that since at least the 1950s. A lot of people who make programming languages use the := for the assignment operator instead.

I too am a Python-preferring machine learning engineer, but my reasons are almost precisely the opposite of the post's.

1. No matter how much you ever think, as a scientist, that you "only do an analysis one time" it is false 99.99999% of the time. You will always want to run it multiple times. Other people will want help modifying and running variations of it. Employers will need you, the scientist, to "productionize" it and make it suitable for automated deployment, probably cross-platform.

2. Your analysis will have to adapt to changing data inputs, which means you invariably have to create a (well-designed, unit-tested, and best-practices compliant) tool kit for custom data cleaning, pre-processing, database I/O, file system I/O, and visualization.

3. You will inevitably need to be concerned with raw-metal performance, but generally in isolated pockets of your code, so you'll need a language like Python that supports targeted performance optimization with tools like Cython.

4. Code is read (especially by newbies who need your help) much more than it is written, so you need a language that is easy to explain and reason about, with very few syntactical tricks and complicated conceptual nuances.

Overall, Python suits this niche very well. It is a full-service object-oriented language with a huge and well-maintained standard library. The third party tools for machine learning and general numeric computing are by far the best in the open source world (apart from a handful of boutique R libraries, which can use via rpy2 anyway), and Python is a simple language that is easy to teach and explain but also supports lots of targeted optimization in the CPython layer.

From the very first line of code you write, when you still naively believe "I will only run this once and I just need to crank it out," you need to be obsessed with writing well-designed, extensible, unit-tested code that is only a short distance from already being "production ready" -- and Python is a great language choice for this.

The reason is that it's more common for a statistician to encounter a problem that needs switching to Python (either because there are Python tools for it, or because of a need to interact with non-statisticians who are using Python), than it is for a software engineer to get into statistical research.

It is quite telling that a lot of people here mention packages such as ggplot2 as the advantage of R -- ggplot2 is really quite awesome, but I can see it being similarly powerful had it been done for Python (maybe with a bit more unwieldy syntax, but that stuff is really subjective).

R really shines when you are doing actual data analysis -- things like the data.frame object, first-class missing values, proper attention paid to inference (something I quite often see missing from stuff written in Python or even Matlab), model syntax, estimation objects with actually useful pretty-printed summary, huge variety of statistical plots, everything designed to be convenient to use in the REPL, etc. -- and since these things are expected, you will usually find them in CRAN packages as well. Python (and Matlab) are IMO much less consistent in that regard.

But if your task is primarily to implement a particular type of analysis, put it in production to be easily repeated in the future, and go to next problem, R indeed offers little benefit over Python unless you rely on a specific package (which, btw, you should only do if you understand exactly what it does, because most R packages are written by statisticians, and as a result have very little fool-proofing).

> 'I believe this'. So far he respects truth

What if he believes is objectively wrong?