R, the master troll of statistical languages (2012) • Hacker News Board

Re: R, the master troll of statistical languages (2012)

#131

Post by klmr » Wed, Feb 17, 2016, 11:25 AM UTC

z92 wrote: post #110

klmr wrote: post #74

Earlier quoted context omitted.

Python and similar dynamic languages suffer from the fact that every name access (variable, function, etc) incurs a dynamic lookup of that name in a (nested) dictionary. Statically compiled languages don’t have this. There are fairly recent, clever optimisations that can avoid many of these lookups but (a) they are not implemented in any of the common implementations of Python, R, etc (JavaScript has them though). Bu…

The lookup thing only happens during compilation to byte code or intermediate code, I believe. Once in byte code, there are no variable names, only addresses.

No, unfortunately that is not the case. Lookup happens at execution of the byte code, because variables cannot be looked up at byte compilation. Consider the following case:

    x = 1
    local({
        assign(user_input, 2)
        print(x)
    }, envir = new.env(parent = environment()))

If `user_input` is “x”, the lookup of `x` in the local scope finds a different variable, in a different scope. Hence this lookup needs to take place every time this piece of code is executed.

I’m not sure if Python suffers from similar problems.

Re: R, the master troll of statistical languages (2012)

#132

Post by klmr » Wed, Feb 17, 2016, 11:34 AM UTC

tachyonbeam wrote: post #130

klmr wrote: post #74

Earlier quoted context omitted.

Python and similar dynamic languages suffer from the fact that every name access (variable, function, etc) incurs a dynamic lookup of that name in a (nested) dictionary. Statically compiled languages don’t have this. There are fairly recent, clever optimisations that can avoid many of these lookups but (a) they are not implemented in any of the common implementations of Python, R, etc (JavaScript has them though). Bu…

As a compiler writer, I can tell you that in JS, local variable lookups do not incur any kind of dynamic overhead. The performance of modern JS engines is much closer to C than you might think. Dynamic language optimization is also not so recent. Most of the techniques implemented by modern JS engines were invented for the Smalltalk and Self projects. See this paper from 1991, for example: http://bibliography.selflan…

I haven’t followed recent development of JavaScript all that closely so my knowledge is somewhat outdated. However, the optimisations that make JS performance close to C in some cases are really recent. Some of the tricks are old, such as the paper you cited. But these tricks only go so far, and in particular even modern GCs simply work badly in memory-constrained environments, which puts a hard upper limit on the amount of memory that JavaScript can handle efficiently. One of the better articles on this subject is [1].

That said, my comment already mentioned that local variable lookup isn’t a problem in JavaScript. It is in R, however; see my example in [2]. Beyond that, both R and Python execution have obvious optimisation potential, which is made hard by the fact that existing libraries rely extensively on implementation details of the current interpreters.

[1] http://sealedabstract.com/rants/why-mobile-web-apps-are-slow...

[2] https://news.ycombinator.com/item?id=11117070

Re: R, the master troll of statistical languages (2012)

#133

Post by grayclhn » Wed, Feb 17, 2016, 11:46 AM UTC

mziel wrote: post #6

louden wrote: post #3

R is a language with a lot of gotcha's. I usually get burned by characters being converted to factors in read.csv() and converting factors to numeric (it works, but not how you intend). The R Inferno ( http://www.burns-stat.com/documents/books/the-r-inferno/ ) has a lot of other gotcha's and is worth a read for people who use the language. That said, the power, flexibility and user community make it my go-to for any…

Factor is one of the worst thing of R-world. I don't recall ever needing factors, yet they creep in with many functions (read.csv, cut). Btw there's a nice readr package (from Hadleyverse) that has a read_csv method that does away with factors by default.

You must not do a lot of regression with categorical data, then. I use commands like `lm(y ~ (x1 + x2) * factor_variable, data = d)` and `xyplot(y ~ x1 | factor_1, groups = factor_2, data = d)` all the time.

Re: R, the master troll of statistical languages (2012)

#134

Post by jcizzle » Wed, Feb 17, 2016, 12:25 PM UTC

jlebar wrote: post #118

mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

> The problem is people using R without trying to learn about the language itself It's not the user's fault. Like, congratulations on being better at R than the author of TFA. Maybe you're smarter than him, maybe you've put in more time learning, maybe you've just spent your time more intelligently, maybe you lucked out and bought better books...who knows. But this line of reasoning completely misses the author's poi…

I think the person you are responding to is simply saying some things are more complex than others and require more understanding and experience. R apparently falls into that category. If the author wants to gain that experience, I think the time spent on this blog post may be better used to reading a book on R. Fault might be a strong word, but the author has certainly made a decision on what they spend their time on and the results are as expected.

Re: R, the master troll of statistical languages (2012)

#135

Post by stewbrew » Wed, Feb 17, 2016, 12:38 PM UTC

klmr wrote: post #78

stewbrew wrote: post #10

Earlier quoted context omitted.

How would you represent categorical data then? R's primary use case isn't text processing. And HW isn't always right.

As character, for instance (in particular, they can do everything factors can do when used in conjunction with `unique`, and sorted factors can be represented as a conjunction of characters and numerics). Factors work better, but only barely. In particular, they are nowadays not any more efficient than using character (!). They used to be, which is why they are liberally used everywhere in R’s base libraries.

"In particular, they are nowadays not any more efficient than using character"

How could a comparison of two strings of unknown size be as efficient as comparing two integers? I'm curious to learn something new.

Re: R, the master troll of statistical languages (2012)

#136

Post by todd8 » Wed, Feb 17, 2016, 1:22 PM UTC

todd8 wrote: post #91

mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

It's been a few years since I really looked at R, but I don't think the problems with R are simply that people don't learn the language. Some languages are simply not as good as others. We can all learn more about the tools we use when programming, I know that I certainly could. But this doesn't make it our fault that a language is tricky or hard to debug or hard to understand. If we worked at it, I suppose we could…

I should point out that Ross Ihaka along with Robert Gentleman created R.

Re: R, the master troll of statistical languages (2012)

#137

Post by baldfat » Wed, Feb 17, 2016, 1:28 PM UTC

jlebar wrote: post #118

mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

> The problem is people using R without trying to learn about the language itself It's not the user's fault. Like, congratulations on being better at R than the author of TFA. Maybe you're smarter than him, maybe you've put in more time learning, maybe you've just spent your time more intelligently, maybe you lucked out and bought better books...who knows. But this line of reasoning completely misses the author's poi…

Its a 4 year old article and R has changed a TON with new things BUT.... R has grown a ton in users as well as features.

R really is a functional programming language that people don't take advantage on. All languages have strength and weaknesses and YET the complaint is R has too many ways to do any one thing which allows us to have data.table, dplyr, ggplot2, magrittr (pipping %>%). [EDIT RStudio and RServer are also a big example of R growth in features and quality]

As I learned R my code has changed dramatically and I think R has one of the largest gap between the code you start with and when you are proficient. My starting R code is really embarrassing.

Re: R, the master troll of statistical languages (2012)

#138

Post by crispyambulance » Wed, Feb 17, 2016, 1:42 PM UTC

mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

I sympathize with the OP and also feel frustrated with R (and I say that as a regular R "practitioner").

Part of the problem, I think, is the built-in documentation. The typical R user is a domain-expert just trying to get some work done. Occasionally, they'll get stuck and try something like "?sapply". What appears is usually a terse, confusing mess that takes a VERY LONG TIME to digest and is the LAST THING you want to read when you're trying to make a living solving a problem other than understanding R documentation.

Below is the "Description" for Apply (which you get when you try ?sapply). Does it _really_ explain the essentials of what you need to use "apply"?

"... lapply returns a list of the same length as X, each element of which is the result of applying FUN to the corresponding element of X.

sapply is a user-friendly version and wrapper of lapply by default returning a vector, matrix or, if simplify = "array", an array if appropriate, by applying simplify2array(). sapply(x, f, simplify = FALSE, USE.NAMES = FALSE) is the same as lapply(x, f).

vapply is similar to sapply, but has a pre-specified type of return value, so it can be safer (and sometimes faster) to use. ..."

Re: R, the master troll of statistical languages (2012)

#139

Post by mziel » Wed, Feb 17, 2016, 1:54 PM UTC

crispyambulance wrote: post #138

mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

I sympathize with the OP and also feel frustrated with R (and I say that as a regular R "practitioner"). Part of the problem, I think, is the built-in documentation. The typical R user is a domain-expert just trying to get some work done. Occasionally, they'll get stuck and try something like "?sapply". What appears is usually a terse, confusing mess that takes a VERY LONG TIME to digest and is the LAST THING you wan…

YMMV, but I think this is pretty clear:

"lapply returns a list of the same length as X, each element of which is the result of applying FUN to the corresponding element of X" == lapply is a map() construct that takes a list and a function

"sapply is a user-friendly version and wrapper of lapply by default returning a vector, matrix or, if simplify = "array", an array if appropriate, by applying simplify2array(). sapply(x, f, simplify = FALSE, USE.NAMES = FALSE) is the same as lapply(x, f)." == "sapply(x, f, simplify = FALSE, USE.NAMES = FALSE) is the same as lapply(x, f)"

Re: R, the master troll of statistical languages (2012)

#140

Post by crispyambulance » Wed, Feb 17, 2016, 2:07 PM UTC

mziel wrote: post #139

crispyambulance wrote: post #138

Earlier quoted context omitted.

I sympathize with the OP and also feel frustrated with R (and I say that as a regular R "practitioner"). Part of the problem, I think, is the built-in documentation. The typical R user is a domain-expert just trying to get some work done. Occasionally, they'll get stuck and try something like "?sapply". What appears is usually a terse, confusing mess that takes a VERY LONG TIME to digest and is the LAST THING you wan…

YMMV, but I think this is pretty clear: "lapply returns a list of the same length as X, each element of which is the result of applying FUN to the corresponding element of X" == lapply is a map() construct that takes a list and a function "sapply is a user-friendly version and wrapper of lapply by default returning a vector, matrix or, if simplify = "array", an array if appropriate, by applying simplify2array(). sapp…

It could be A LOT more clear. The first sentence is, of course, obvious to everyone and probably already known by people searching for help on sapply.

If you have a family of functions that "sort-of" do similar things, the most critical thing to communicate is some clear sense of when to use one or another of the functions. This doc degenerates into unhelpful jibberish instead.

Perhaps a close reading of it would have helped the OP, but there is a unnecessarily high cost in frustration.