Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R, the master troll of statistical languages (2012)

talyarkoni.org

Post reply on HN
Print view View original

121–130 of 154 posts

Re: R, the master troll of statistical languages (2012)

#121

Post by sdenton4 »

GFK_of_xmaspast wrote: post #69
existencebox wrote: post #33

Earlier quoted context omitted.

It's funny you bring up python; I say this not as a comment on your thesis, but related, since I often hear the "python is slow" trope but that's only half true, you can typically write python that is plenty "fast enough" (As a day-jobbing data pipeline engineer) if you're implementing with an understanding of what things will drive you into the mud. This goes beyond just understanding the tool you're using, fundamen…

I've translated plenty of numerical code from (pure-ish) python to c and c++, and usually get about a 100x speedup, sometimes as high as 800x, implementing the same algorithms.

I've had good experience with Cython, which compiles python to C and gets almost all of the speedup of rewriting in C entirely. And in fact, most of that speedup just comes from declaring variable types...

Re: R, the master troll of statistical languages (2012)

#122

Post by TheLogothete »

th0ma5 wrote: post #23

My own personal rant, I think the specific feeling I get is the conceptual idea of R has long since outpaced the reality of R. People like to fetishize data, and R sure lets you do that. The data science landscape however is growing such that R is really just a one-trick pony, however, that one trick is for better or worse being the gold standard of statistics and modeling, somehow. But everything else wants to sugar…

Except that you miss a very important point - nobody cares about the stuff you listed. I build models and use shiny to create a front end for clients to interact with them. They are very happy and pay me very handsomely. I can assure you that this is the case across the board. R is for analysts, not for programmers. It seems like programmers feel intimidated, because analysts now code their solutions themselves.

Re: R, the master troll of statistical languages (2012)

#123

Post by TheLogothete »

elcapitan wrote: post #29

Not intending to start a language war here, but if somebody who has experience with both R and Python/pandas/etc could answer - how's the current state of the emerging Python data/statistics ecosystem compared to R? (not counting all the other differences like R being allegedly weird or Python more general purpose and so on).

We did an evaluation recently. Not even close.

Re: R, the master troll of statistical languages (2012)

#124

Post by dmichulke »

Fomite wrote: post #39
x0x0 wrote: post #12

Earlier quoted context omitted.

It's kind of amazing to see someone admit to spending hundreds or thousands of hours using R, yet refuse to spend a couple hours learning the language a little better. Whining that your tools are hard without investing any effort in them is just dumb. The R help even comes with code samples that you can run!

R is, I think, an interesting language because it's heavily used by people who would not otherwise learn a programming language. If you compare R not with other programming languages, but with other ways of working with statistical data, this makes far more sense. I don't actually "know" SAS in the way I know a programming language - I know the commands I invoke to do what I want it to do. Similarly, I encounter lots…

My personal experience is similar because I know quite a few people in social sciences.

Conceptually, this is similar to rats pulling a lever or monkeys being reinforced to type the right characters. It also explains p-hacking and many other problems of interpretation.

Now one question I always have is - if you consider R just a tool - what is the difference between things I should fully understand (R?) and things I should only know how to use (e.g., my cell phone)?

How can I justify saying that people should understand R while I myself don't understand quite a few aspects of my cell phone?

Re: R, the master troll of statistical languages (2012)

#126

Post by simonbyrne »

Fomite wrote: post #39
x0x0 wrote: post #12

Earlier quoted context omitted.

It's kind of amazing to see someone admit to spending hundreds or thousands of hours using R, yet refuse to spend a couple hours learning the language a little better. Whining that your tools are hard without investing any effort in them is just dumb. The R help even comes with code samples that you can run!

R is, I think, an interesting language because it's heavily used by people who would not otherwise learn a programming language. If you compare R not with other programming languages, but with other ways of working with statistical data, this makes far more sense. I don't actually "know" SAS in the way I know a programming language - I know the commands I invoke to do what I want it to do. Similarly, I encounter lots…

Also, many people use it only intermittently, maybe once every six months or so when they have some data to look at. Rather than try to relearn the language and its quirks yet again, it's much easier to take what you did last time and tweak it until you get what you need.

Re: R, the master troll of statistical languages (2012)

#127

Post by hadley »

capnrefsmmat wrote: post #68

My biggest complaint about R isn't the inconsistency and obtuseness -- I've been using it long enough to get familiar with the documentation and the zillions of varieties of apply. My problem is the data structures. R has only a few core data structures: vectors, lists, arrays, and matrices. Data frames are built on top of lists, and admittedly data frames are incredibly useful for statistics -- there's a reason pand…

The lack of data structures in R is a totally fixable problem.

Re: R, the master troll of statistical languages (2012)

#128

Post by elcapitan »

TheLogothete wrote: post #123
elcapitan wrote: post #29

Not intending to start a language war here, but if somebody who has experience with both R and Python/pandas/etc could answer - how's the current state of the emerging Python data/statistics ecosystem compared to R? (not counting all the other differences like R being allegedly weird or Python more general purpose and so on).

We did an evaluation recently. Not even close.

Is that evaluation somehow public, or could you share some details?

Re: R, the master troll of statistical languages (2012)

#129

Post by YeGoblynQueenne »

sampo wrote: post #115
YeGoblynQueenne wrote: post #38

Earlier quoted context omitted.

>> But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions? R is interpreted and dynamically typed, so when you declare a variable, the interpreter has to do some bookkeeping to figure out the type of the variable, allocate memory for it and so on. If you write a loop by hand, the interpreter has to do this bookkeeping once for each iteration.…

> If we want a language with fast loops we have to rely on C or Fortran and forget about vectorised notation. Fortran (Fortran 90 specifically) got vector notation 20 years ago.

I suspected this might be the case but I don't know Fortran. Maybe you're right about Julia and Lua also, I'll have to investigate.

Re: R, the master troll of statistical languages (2012)

#130

Post by tachyonbeam »

klmr wrote: post #74
Cogito wrote: post #71

Earlier quoted context omitted.

Any idea where the speedups came from? Is it that the problems weren't algorithmically limited in the first place (lots of io for example), reduction of overhead etc (what kind of python was the code running on before?), or just that the speedup on low level operations added up cumulatively and cam e to dominate the other timing factors? Also, did you change the data structures or use the same ones as in python? Was…

Python and similar dynamic languages suffer from the fact that every name access (variable, function, etc) incurs a dynamic lookup of that name in a (nested) dictionary. Statically compiled languages don’t have this. There are fairly recent, clever optimisations that can avoid many of these lookups but (a) they are not implemented in any of the common implementations of Python, R, etc (JavaScript has them though). Bu…

As a compiler writer, I can tell you that in JS, local variable lookups do not incur any kind of dynamic overhead. The performance of modern JS engines is much closer to C than you might think. Dynamic language optimization is also not so recent. Most of the techniques implemented by modern JS engines were invented for the Smalltalk and Self projects. See this paper from 1991, for example: http://bibliography.selflanguage.org/_static/implementation....

Python is just inexcusably non-optimized. It's a bytecode interpreter, with each instruction requiring dynamic dispatch. Integers are represented using actual objects, with pointer indirection. The most naive, non-optimizing JIT implementation might get you a 10x speedup over CPython. I think that eventually, as better-optimised dynamic languages gain popularity, people will come to accept that there is no excuse for dynamic language implementations to perform this poorly.

Post reply on HN