Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R, the master troll of statistical languages (2012)

talyarkoni.org

Post reply on HN
Print view View original

41–50 of 154 posts

Re: R, the master troll of statistical languages (2012)

#41

Post by chrisseaton »

YeGoblynQueenne wrote: post #38
chrisseaton wrote: post #25

Earlier quoted context omitted.

> For example complaining that R is slow and then writing iterative solution instead of using vectorisation. But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions?

>> But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions? R is interpreted and dynamically typed, so when you declare a variable, the interpreter has to do some bookkeeping to figure out the type of the variable, allocate memory for it and so on. If you write a loop by hand, the interpreter has to do this bookkeeping once for each iteration.…

Why can't a JIT solve this? It shouldn't need to do the bookkeeping for every iteration if it has JIT compiled it. A JIT should be able to take advantage of processor vector instructions etc.

Re: R, the master troll of statistical languages (2012)

#42

Post by copperx »

x0x0 wrote: post #12
mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

It's kind of amazing to see someone admit to spending hundreds or thousands of hours using R, yet refuse to spend a couple hours learning the language a little better. Whining that your tools are hard without investing any effort in them is just dumb. The R help even comes with code samples that you can run!

Surprisingly, this is true for many programmers of general-purpose languages.

Re: R, the master troll of statistical languages (2012)

#43

Post by Fomite »

elcapitan wrote: post #29

Not intending to start a language war here, but if somebody who has experience with both R and Python/pandas/etc could answer - how's the current state of the emerging Python data/statistics ecosystem compared to R? (not counting all the other differences like R being allegedly weird or Python more general purpose and so on).

Also assuming you don't want to count things like "You can call R from Python, and Python from R", here's my take on it, as someone who uses both languages:

- Pandas has helped Python tremendously, but I don't think it's quite to where the R data frame is.

- For 90% of what someone who wants to do statistics wants to do, it honestly doesn't matter at all. You can do nice data visualization in both. You can fit most generalized linear models in both.

- At the cutting edge, R still takes the cake. Odds are if someone has developed a new method (especially outside machine learning), it's in R before it's in Python. Your local university's statistics department is likely running R (or SAS), not Python.

Re: R, the master troll of statistical languages (2012)

#44

Post by x0x0 »

chrisseaton wrote: post #41
YeGoblynQueenne wrote: post #38

Earlier quoted context omitted.

>> But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions? R is interpreted and dynamically typed, so when you declare a variable, the interpreter has to do some bookkeeping to figure out the type of the variable, allocate memory for it and so on. If you write a loop by hand, the interpreter has to do this bookkeeping once for each iteration.…

Why can't a JIT solve this? It shouldn't need to do the bookkeeping for every iteration if it has JIT compiled it. A JIT should be able to take advantage of processor vector instructions etc.

There's some movement in that direction.

However, the R core committers are essentially not only volunteers, but they're all (afaik) academic statisticians. One of the people who made strides in this direction is primarily an computational statistician at Iowa (Luke Tierney / compiler package). Building a high performance runtime/jit is wildly out of their scope of expertise.

In retrospect, and I think many of them would agree, building and maintaining their own runtime was a giant mistake. Yet here we are.

Serious compiler people (Jan Vitek, others) have made strides towards a faster implementation (his in java / fastr IIRC), but it suffers from the same problem as cpython: there are millions of lines of C code in packages or internal functions that have the details of the R interpreter / C interface deeply embedded in them. In fact, there's probably far more "R" code written in C than in R. Undoing this mess is not easy, and probably not possible.

Oh, reading Evaluating the Design of the R Language [1] will shed some more light on why it's hard to make R run fast.

[1] http://r.cs.purdue.edu/pub/ecoop12.pdf

edited to correctly describe Luke as per gbrown

Re: R, the master troll of statistical languages (2012)

#45

Post by gbrown »

chrisseaton wrote: post #41
YeGoblynQueenne wrote: post #38

Earlier quoted context omitted.

>> But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions? R is interpreted and dynamically typed, so when you declare a variable, the interpreter has to do some bookkeeping to figure out the type of the variable, allocate memory for it and so on. If you write a loop by hand, the interpreter has to do this bookkeeping once for each iteration.…

Why can't a JIT solve this? It shouldn't need to do the bookkeeping for every iteration if it has JIT compiled it. A JIT should be able to take advantage of processor vector instructions etc.

R does actually ship with the ability to byte-compile functions these days, and as that functionality matures it may become the default behavior. It's still better to actually learn the language; it's far easier to optimize something like:

    apply(X, 1, function(x){
        # do stuff to the row of X
    })
than:

    for (i in 1:nrow(X)){
        # do stuff to X[i,], and store it somewhere
    }

Re: R, the master troll of statistical languages (2012)

#46

Post by Bluestrike2 »

I just introduced a friend of mine to R. He's working on his PhD in microbiology and beside himself once he started working with R. Personally, I can't believe he hadn't used it before. It really is a beautiful language to work with once you get a handle on it.

Re: R, the master troll of statistical languages (2012)

#47

Post by rcthompson »

Maybe R should have optional "training wheels" that produce a warning every time an implicit conversion happens. In the OP's case, it would warn that a data frame was implicitly being converted to a matrix, and maybe also warn that the numeric vectors within were being converted to character in order to get slotted into that matrix.

Re: R, the master troll of statistical languages (2012)

#48

Post by nathell »

I have a love-hate relationship with R, being a predominantly Clojure (and Ruby these days) programmer who only occassionally dabbles in data crunching.

The apply/sapply dichotomy that the article mentions (actually a hexachotomy, there are also mapply, sapply, tapply and vapply) is one example of a gazillion warts that the language has.

Another random one: R has a useful function, paste, that concatenates strings together. Only it takes varargs, not a character vector, so if you have a vector v of strings, you have to use do.call(paste, v). Only not, because do.call insists that its second argument be a list, not a vector, so you do do.call(paste, as.list(v)). And if you want to separate the strings, say, by commas, you have to affix the named argument sep, obtaining do.call(paste, c(as.list(v), sep=",")).

And R's three mutually incompatible object systems. And so on and so on and so on.

There are things to love. The packaging system works really well. I like the focus that R puts on documentation: hardly anywhere is it so comprehensive, with vignettes and all. There are things plainly inspired by Lisp (R is just about the only non-Lisp I know that has a condition and restart system akin to CL). And ggplot2 is one hell of a gem of API.

In many ways, R is the PHP of data science. (Though the core language's still nowhere near as abysmal as PHP.) Despite all the warts, there are all sorts of statistical analyses that are just a package.install() away. Put another way, R is to data science what LaTeX is to typesetting. It's a heavy pile of ducttape, but it's here to stay because it's just so damn useful.

Re: R, the master troll of statistical languages (2012)

#49

Post by gbrown »

x0x0 wrote: post #44
chrisseaton wrote: post #41

Earlier quoted context omitted.

Why can't a JIT solve this? It shouldn't need to do the bookkeeping for every iteration if it has JIT compiled it. A JIT should be able to take advantage of processor vector instructions etc.

There's some movement in that direction. However, the R core committers are essentially not only volunteers, but they're all (afaik) academic statisticians. One of the people who made strides in this direction is primarily an computational statistician at Iowa (Luke Tierney / compiler package). Building a high performance runtime/jit is wildly out of their scope of expertise. In retrospect, and I think many of them w…

I don't think calling Luke an "agricultural statistician" is at all reflective of his work. Not everything in Iowa is corn, and Luke has been working in computationally intensive statistical methodology and statistical software development for decades.

Re: R, the master troll of statistical languages (2012)

#50

Post by bachmeier »

mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

> The problem is people using R without trying to learn about the language itself

I see a lot of blub when I read posts about R. So much so that I start with the assumption that any post about R is a blub post.

Post reply on HN