Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R, the master troll of statistical languages (2012)

talyarkoni.org

Post reply on HN
Print view View original

111–120 of 154 posts

Re: R, the master troll of statistical languages (2012)

#111

Post by z92 »

klmr wrote: post #74
Cogito wrote: post #71

Earlier quoted context omitted.

Any idea where the speedups came from? Is it that the problems weren't algorithmically limited in the first place (lots of io for example), reduction of overhead etc (what kind of python was the code running on before?), or just that the speedup on low level operations added up cumulatively and cam e to dominate the other timing factors? Also, did you change the data structures or use the same ones as in python? Was…

Python and similar dynamic languages suffer from the fact that every name access (variable, function, etc) incurs a dynamic lookup of that name in a (nested) dictionary. Statically compiled languages don’t have this. There are fairly recent, clever optimisations that can avoid many of these lookups but (a) they are not implemented in any of the common implementations of Python, R, etc (JavaScript has them though). Bu…

The lookup thing only happens during compilation to byte code or intermediate code, I believe. Once in VM, there are no variable names. Only addresses.

Re: R, the master troll of statistical languages (2012)

#112

Post by jghn »

avs733 wrote: post #86
wanderfowl wrote: post #32

I'm a Post-Doc in a small social sciences department in a major university, and am probably the department's ranking R-geek. I did my dissertation, and much of my current work, doing modeling, analysis, and even machine learning in R. In many ways, I owe much of my success to the power that R has allowed me to wield. Multicore lapplys and ggplot2 are my life these days. But even with this, R drives me absolutely batt…

I am in an almost identical situation, just still in the PhD stage. I moved from Matlab (originally a mechanical engineer) and the biggest shock was documentation and just the internal help stuff in general. The help files on a regular basis require you to understand how something works to understand the thing explaining how it works. I am often just shocked at little quirks I find trying to do things in R, not that…

Iirc it comes from the old APL keyboard

Re: R, the master troll of statistical languages (2012)

#113

Post by snaky »

jghn wrote: post #112
avs733 wrote: post #86

Earlier quoted context omitted.

I am in an almost identical situation, just still in the PhD stage. I moved from Matlab (originally a mechanical engineer) and the biggest shock was documentation and just the internal help stuff in general. The help files on a regular basis require you to understand how something works to understand the thing explaining how it works. I am often just shocked at little quirks I find trying to do things in R, not that…

Iirc it comes from the old APL keyboard

That may be not a bad idea to get an APL keyboard actually, considering there are R interop interfaces for DyalogAPL, APLX and J.

Re: R, the master troll of statistical languages (2012)

#114

Post by kgwgk »

tmalsburg2 wrote: post #62

Writing a variant of this article has become a rite of passage for all serious users of R. There are two issues that contribute to the difficulties people experience with R. First, yes, R can be confusing at times. Tal explains this really well, but only scratches the surface. There is so much more confusing and counter-intuitive stuff, for example with regards to factors that only very few people seem to understand…

There are other books on R programming: John Chambers published "Programming with Data: A Guide to the S Language" in 1998 and "Software for Data Analysis: Programming with R" in 2008.

Re: R, the master troll of statistical languages (2012)

#115

Post by sampo »

YeGoblynQueenne wrote: post #38
chrisseaton wrote: post #25

Earlier quoted context omitted.

> For example complaining that R is slow and then writing iterative solution instead of using vectorisation. But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions?

>> But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions? R is interpreted and dynamically typed, so when you declare a variable, the interpreter has to do some bookkeeping to figure out the type of the variable, allocate memory for it and so on. If you write a loop by hand, the interpreter has to do this bookkeeping once for each iteration.…

> If we want a language with fast loops we have to rely on C or Fortran and forget about vectorised notation.

Fortran (Fortran 90 specifically) got vector notation 20 years ago.

Re: R, the master troll of statistical languages (2012)

#116

Post by sampo »

YeGoblynQueenne wrote: post #38
chrisseaton wrote: post #25

Earlier quoted context omitted.

> For example complaining that R is slow and then writing iterative solution instead of using vectorisation. But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions?

>> But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions? R is interpreted and dynamically typed, so when you declare a variable, the interpreter has to do some bookkeeping to figure out the type of the variable, allocate memory for it and so on. If you write a loop by hand, the interpreter has to do this bookkeeping once for each iteration.…

> If we want an interpreted language with somewhat intuitive notation, then it has to have crappy slow loops.

Unless you're Julia, JavaScript or Lua with a fiendishly clever virtual machine. Look at the benchmark figure here: http://julialang.org/

Re: R, the master troll of statistical languages (2012)

#117

Post by mikeskim »

avs733 wrote: post #89
makeset wrote: post #76

Yes, I'm well aware of R's many faults, I have my own long list of R caveats I hand to new hires, but not bothering to learn the damn language is no reason to complain about it. First, RTFM.

look, I am all for RTFM but my experience with R has been a different case in my experience. Can I make it do powerful things? yes. can I take code that wont run, copy it to a new file and have it run? yes. 1) There are just quirks in implementation that are nonintuitive. I constantly find myself doing things I would do in other languages, to do simple tasks, which fail for no good reason, albeit an obvious one once…

You wouldn't find it intuitive if your first language was Scheme.

Re: R, the master troll of statistical languages (2012)

#118

Post by jlebar »

mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

> The problem is people using R without trying to learn about the language itself

It's not the user's fault.

Like, congratulations on being better at R than the author of TFA. Maybe you're smarter than him, maybe you've put in more time learning, maybe you've just spent your time more intelligently, maybe you lucked out and bought better books...who knows.

But this line of reasoning completely misses the author's point, which is that despite having used the language for years, he still finds it inscrutable. "It would be easier if you were better at R" is a tautology, and unhelpful. The issue is that the author finds it hard to become better at R.

We can disagree as to whether or not it's objectively hard to become better at R, but this is a perfectly valid criticism to make. It's not the user's fault.

Re: R, the master troll of statistical languages (2012)

#119

Post by TheLogothete »

todd8 wrote: post #91
mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

It's been a few years since I really looked at R, but I don't think the problems with R are simply that people don't learn the language. Some languages are simply not as good as others. We can all learn more about the tools we use when programming, I know that I certainly could. But this doesn't make it our fault that a language is tricky or hard to debug or hard to understand. If we worked at it, I suppose we could…

The push for R is enormous. It's not getting replaced.

Re: R, the master troll of statistical languages (2012)

#120

Post by sdenton4 »

todd8 wrote: post #91
mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

It's been a few years since I really looked at R, but I don't think the problems with R are simply that people don't learn the language. Some languages are simply not as good as others. We can all learn more about the tools we use when programming, I know that I certainly could. But this doesn't make it our fault that a language is tricky or hard to debug or hard to understand. If we worked at it, I suppose we could…

From your link, hilariously relevant to the blog post at hand:

"First, scalar computations in R are very slow. This in part because the R interpreter is very slow, but also because there are a no scalar types. By introducing scalars and using compilation it looks like its possible to get a speedup by a factor of several hundred for scalar computations. This is important because it means that many ghastly uses of array operations and the apply functions could be replaced by simple loops. The cost of these improvements is that scope declarations become mandatory and (optional) type declarations are necessary to help the compiler."

Post reply on HN