Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

R, the master troll of statistical languages (2012)

talyarkoni.org

Post reply on HN
Print view View original

31–40 of 154 posts

Re: R, the master troll of statistical languages (2012)

#31

Post by th0ma5 »

JasonCEC wrote: post #27
th0ma5 wrote: post #23

My own personal rant, I think the specific feeling I get is the conceptual idea of R has long since outpaced the reality of R. People like to fetishize data, and R sure lets you do that. The data science landscape however is growing such that R is really just a one-trick pony, however, that one trick is for better or worse being the gold standard of statistics and modeling, somehow. But everything else wants to sugar…

I disagree on multiple fronts! 1) ggplot does exactly what it is supposed to do: create data visualizations. It made no promises for interactivity or display, and in fact, it was originally designed for creating publication quality charts, which it continues to do well. 1.5) ggvis is a D3 API wrapper on ggplot and allows for interactive graphics. Do you want to pay your data scientists for creating production ready g…

I sincerely appreciate your thorough and well reasoned response, and thank you for taking the time with it.

1) I agree, perhaps I was trying to allude to visualizations being more than charts. ggplot's charts are absolutely gorgeous and simpler to make than even I remember them being in Lotus 123 for DOS. However, there are some things in the periodic table of visualizations (http://www.visual-literacy.org/periodic_table/periodic_table...) that it can't do. And what about hybrid combinations in the same chart? Could I have a bar chart where the bars are also mini-spectrograms? I can instantly think of how to do this in SVG or Processing, but I'm not sure where to begin with ggplot ... maybe it is possible. Of course why would you?

1.5) I guess I don't want to pay someone else to do the custom thing in D3 that the statistician can almost do with their code, or try to get a regular web stack developer familiar with D3 to actually get it working the way the statistician says.

2) Yup, totally agree!

3) I think Shiny takes a lot of liberties and makes a lot of assumptions that users of it can't even express to me are important to them because using Shiny completely hides the underlying concepts of how it is implemented. I guess I would definitely call that magic. You're right there's quite a lack of magic in most of the language and packages, however.

4) I think data science can/should/does embrace the web. I think the modeling stack shouldn't be on the application server, but a trained model perhaps should be? I also wouldn't trust the stability of R for performance critical API calls without a lot of redundant instances and a lot of load balancing.

Anyway... the real problem is that you're also absolutely right. There is quite a bit difference in opinion between the tooling of an analysis effort, and the robustness expected by IT.

Thanks again!

Re: R, the master troll of statistical languages (2012)

#32

Post by wanderfowl »

I'm a Post-Doc in a small social sciences department in a major university, and am probably the department's ranking R-geek. I did my dissertation, and much of my current work, doing modeling, analysis, and even machine learning in R.

In many ways, I owe much of my success to the power that R has allowed me to wield. Multicore lapplys and ggplot2 are my life these days. But even with this, R drives me absolutely batty, and the documentation, even battier.

I may be competent relative to most, but R feels so taped-together and idiosyncratic that even on my best days, I just feel like a newbie who's built up an army of ugly hacks.

Someday, I'll learn more about the python stats tools and do my stats there. But for now, R it is. Troll on, you crazy bastard.

Re: R, the master troll of statistical languages (2012)

#33

Post by existencebox »

TheLogothete wrote: post #16
mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

In my experience a lot of the claims that R is slow are greatly exaggerated and made by people who don't actually use it. Kind of an echo chamber. Every time I see someone say chose Python instead because of speed, I roll my eyes.

It's funny you bring up python; I say this not as a comment on your thesis, but related, since I often hear the "python is slow" trope but that's only half true, you can typically write python that is plenty "fast enough" (As a day-jobbing data pipeline engineer) if you're implementing with an understanding of what things will drive you into the mud. This goes beyond just understanding the tool you're using, fundamentally writing an O(N^N) or something is going to hurt even if you're in C#/C. Have seen that plenty, frankly, more than I've seen "legitimately slow python"

Anyway, this was just a thought rolling around in my head given the discussion.

Re: R, the master troll of statistical languages (2012)

#34

Post by joelberman »

I think if you are a programmer or have some programming language experience, R is not very weird. But if you are a financial analyst or a social scientist, or a statistician, and only want to get your work done; it depends on your first programming language. If it was S3 you are golden. If it was Basic, you are not so golden. Mine was LISP.

Re: R, the master troll of statistical languages (2012)

#35

Post by kbenson »

mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

> The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language.

I think you just explained Perl in a nutshell (err, the idiom, not the book). It seems whenever a language supports enough idioms of the usual C-like languages people will gravitate towards those, likely due to the high population of people that know those idioms and can fall-back on them without having to think too hard. I doubt Lisp has as much of a problem of people trying to write C in Lisp.

Re: R, the master troll of statistical languages (2012)

#36

Post by [deleted] »

chrisseaton wrote: post #25
mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

> For example complaining that R is slow and then writing iterative solution instead of using vectorisation. But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions?

[deleted]

Re: R, the master troll of statistical languages (2012)

#37

Post by Fomite »

TheLogothete wrote: post #16
mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

In my experience a lot of the claims that R is slow are greatly exaggerated and made by people who don't actually use it. Kind of an echo chamber. Every time I see someone say chose Python instead because of speed, I roll my eyes.

My usual take on this: "Between R and Python, the faster language is likely whichever library author actually wrote most of their code in C or FORTRAN."

Re: R, the master troll of statistical languages (2012)

#38

Post by YeGoblynQueenne »

chrisseaton wrote: post #25
mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

> For example complaining that R is slow and then writing iterative solution instead of using vectorisation. But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions?

>> But if someone prefers iterative solutions, or that's all they know, why can't R make them just as fast as the vectorised versions?

R is interpreted and dynamically typed, so when you declare a variable, the interpreter has to do some bookkeeping to figure out the type of the variable, allocate memory for it and so on.

If you write a loop by hand, the interpreter has to do this bookkeeping once for each iteration.

If you write your code in vectorised form, the interpreter can sort out the bookkeeping once and then hand over to the lower-level code (C or Fortran) the vectorised functions are interpreted in.

This can also be further optimised to take advantage of processor vector instructions, parallel processing etc.

So I'm afraid we can't have our pie and eat it. If we want an interpreted language with somewhat intuitive notation, then it has to have crappy slow loops. If we want a language with fast loops we have to rely on C or Fortran and forget about vectorised notation.

Re: R, the master troll of statistical languages (2012)

#39

Post by Fomite »

x0x0 wrote: post #12
mziel wrote: post #4

The problem is people using R without trying to learn about the language itself, just assuming it works like their favourite language. For example complaining that R is slow and then writing iterative solution instead of using vectorization. When I saw the example the author gave my first thought was "sapply/lapply". Lapply is essential to the R use, and is being taught early on in every book/course on R I've ever sa…

It's kind of amazing to see someone admit to spending hundreds or thousands of hours using R, yet refuse to spend a couple hours learning the language a little better. Whining that your tools are hard without investing any effort in them is just dumb. The R help even comes with code samples that you can run!

R is, I think, an interesting language because it's heavily used by people who would not otherwise learn a programming language. If you compare R not with other programming languages, but with other ways of working with statistical data, this makes far more sense. I don't actually "know" SAS in the way I know a programming language - I know the commands I invoke to do what I want it to do.

Similarly, I encounter lots of people using R who don't actually know what a function is, just that lm(x~y) gets them what they want.

Re: R, the master troll of statistical languages (2012)

#40

Post by Fomite »

mziel wrote: post #6
louden wrote: post #3

R is a language with a lot of gotcha's. I usually get burned by characters being converted to factors in read.csv() and converting factors to numeric (it works, but not how you intend). The R Inferno ( http://www.burns-stat.com/documents/books/the-r-inferno/ ) has a lot of other gotcha's and is worth a read for people who use the language. That said, the power, flexibility and user community make it my go-to for any…

Factor is one of the worst thing of R-world. I don't recall ever needing factors, yet they creep in with many functions (read.csv, cut). Btw there's a nice readr package (from Hadleyverse) that has a read_csv method that does away with factors by default.

I actually use factors a fair amount, and having factor-like data shoved into numeric values gets you to some bad places statistically.
Post reply on HN