An opinionated view of the Tidyverse “dialect” of the R language • Hacker News Board

Re: An opinionated view of the Tidyverse “dialect” of the R language

#91

Post by ellisv » Fri, Jul 05, 2019, 8:41 PM UTC

As an R user of some 10+ years, I think some of this is spot on and some of it is just silly.

Pretty much everyone knows the Tidyverse is slow to run, but many of us find it faster to write and very readable because it avoids one-time-use assignments (i.e. in `a need the Tidyverse and it doesn't run faster. Fair criticism.

> The "star" of the Tidyverse, dplyr, consists of 263 functions. While a user initially need not use more than a small fraction of them, the high complexity is clear. Every time a user needs some variant of an operation, she must sift through those hundreds of functions for one suited to her current need.

> By contrast, if she knows base-R (not difficult), she can handle any situation.

Yet base R has some 1,346 functions. Another 455 functions in `stats`. I'm not sure the quantity of functions has as much bearing on learning as the quality and consistency. So this seems like a silly argument to me.

Mostly I think Prof. Matloff is forgetting that base R is kinda terrible and no one else has done much about it aside from RStudio.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#92

Post by mbreese » Fri, Jul 05, 2019, 8:45 PM UTC

lottin wrote: post #40

nerdponx wrote: post #25

Earlier quoted context omitted.

The R standard library is kind of weird and hard to use for "general purpose" programming. It's very clearly a domain specific language. Also, it's unbelievably slow for basic operations like looping, function calls, and variable assignment. It's literally orders of magnitude slower than Python (I've tested it). Unlike its fellow C-flavored-Lisp Javascript, R retains an extreme level of homoiconicity, Which apparentl…

To say that R is a domain-specific language is hardly a criticism, though. It was designed to be domain-specific from the start.

I’d say one of the major problems I have with R is when people try to use it as a general purpose language. Just because you can doesn’t mean you should. It just wasn’t designed that way. I’ve seen a lot of R code that would have been trivial to write and execute significantly faster in another language.

But for its domain, I think R is great.

I guess when all you have is a hammer...

Re: An opinionated view of the Tidyverse “dialect” of the R language

#93

Post by vharuck » Fri, Jul 05, 2019, 8:59 PM UTC

atheriel wrote: post #31

Most of these tidyverse vs. data.table arguments end up sounding a bit irrelevant to me. They tend to focus on syntax, which is largely down to preference, or performance, which is really only important once in a while. And they seem oddly focused on the need to choose one or the other, when it seems obvious that all are good choices with various tradeoffs. I tried my hand at saying something more original about this…

It's not that RStudio has nefarious motives that cause harm, it's that they have neutral and sometimes benevolent motives with unforeseen consequences.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#94

Post by cwyers » Fri, Jul 05, 2019, 9:24 PM UTC

ellisv wrote: post #91

As an R user of some 10+ years, I think some of this is spot on and some of it is just silly. Pretty much everyone knows the Tidyverse is slow to run, but many of us find it faster to write and very readable because it avoids one-time-use assignments (i.e. in `a need the Tidyverse and it doesn't run faster. Fair criticism. > The "star" of the Tidyverse, dplyr, consists of 263 functions. While a user initially need no…

> Mostly I think Prof. Matloff is forgetting that base R is kinda terrible and no one else has done much about it aside from RStudio.

I think he'd disagree with the contention that base R is kinda terrible. Like, I'm pretty sure he's wrong. But this isn't something he's forgotten, it's something he's never believed.

The point about the Tidyverse and RStudio is that RStudio doesn't make any of its revenue from the Tidyverse. They distribute it free. There are no upsells for any of it -- I can't go out and buy ggplot2 add ons, I can't buy dplyr support contracts. There's no money to be made on the Tidyverse, and plenty of resources thrown at it.

So what is RStudio selling? They sell an IDE, they sell servers to host things built in R, they sell a package manager solution, they offer most of these things as a SaaS offering as well (although rstudio.cloud is still in trial and they haven't started charging yet).

In other words, RStudio offers tooling for people who write R. They don't really care what kind of R you're writing, from a financial perspective. They get paid the same if you're using RStudio to write stuff using data.table as they get paid if you use dplyr. RStudio is making a bet that more people will want to write R code if they make R a more pleasant language to work in. And anecdotally, they're absolutely right -- I tried to learn R before the Tidyverse, and I hated every moment of it. I like post-Tidyverse R (although I got into it when people still called it the Hadleyverse). I know several other people with similar experiences to mine.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#95

Post by ahurmazda » Fri, Jul 05, 2019, 9:24 PM UTC

With respect, its laughable to think that anyone working on big-data with R has _not_ heard of `data.table`. When your livelihood depends on it, you tend to find it out one way or the other.

However, I'd claim most folks don't deal with big data or even medium data! [Aside: I chuckled at the 1e5 columns on the plot]. Personally, at the small-to-medium range, all languages are fast enough. It boils down to ergonomics for me. I _detest_ the subscript notation. It slows me down -- not just productivity-wise but also how I read or reason-about my code. Not surprisingly, I prefer `dplyr` over `data.table` (base R or `pandas`)

Re: An opinionated view of the Tidyverse “dialect” of the R language

#96

Post by cwyers » Fri, Jul 05, 2019, 9:30 PM UTC

kgwgk wrote: post #87

minimaxir wrote: post #86

Earlier quoted context omitted.

That's an interesting caveat with pipes/magrittr (I didn't know of it), although it's out of scope of this debate since this discussion is just about how tidyverse uses pipes. In my experience if there's a package that's required in a script/R Notebook outside of tidyverse, I just import the entire library at the start of the file Python-style (to make the dependency obvious), which would avoid this issue.

If you claim that something like the_data x) the_data is a pain to read compared to the_data % subset(variable_a > x) %>% transform(variable_c = variable_a/variable_b) %>% head(100) I thinks it’s fair to mention how by making things more complicated by using pipes you expose yourself to other issues. (In my opinion the first variant is not less readable, and it has the advantage of allowing any of the operations to b…

Because the pipes are at the end of the line, you can comment out a line and it will still work fine unless you're removing the last line of the chain.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#97

Post by kgwgk » Fri, Jul 05, 2019, 9:35 PM UTC

cwyers wrote: post #96

kgwgk wrote: post #87

Earlier quoted context omitted.

If you claim that something like the_data x) the_data is a pain to read compared to the_data % subset(variable_a > x) %>% transform(variable_c = variable_a/variable_b) %>% head(100) I thinks it’s fair to mention how by making things more complicated by using pipes you expose yourself to other issues. (In my opinion the first variant is not less readable, and it has the advantage of allowing any of the operations to b…

Because the pipes are at the end of the line, you can comment out a line and it will still work fine unless you're removing the last line of the chain.

That’s precisely what I said.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#98

Post by cwyers » Fri, Jul 05, 2019, 9:47 PM UTC

notafraudster wrote: post #83

robust wrote: post #80

Earlier quoted context omitted.

Can you provide an example of a problem that requires multiple tidyverse operations, but could be solved equally well using only lapply?

I run a large scale national survey. We download the data from our survey platform. Survey respondents are asked 100+ questions. The questions change week to week and so the column names are not consistent. We exclude respondents who appear to be cheating the system (rushing through questions, straight-lining, skipping almost every question, etc.) As part of our completion check, we want to do a row-wise map of the d…

map() in purrr is functionally equivalent to lapply(). If you can do something in lapply, you can do it with map.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#99

Post by lispm » Fri, Jul 05, 2019, 9:47 PM UTC

vincent-toups wrote: post #75

nerdponx wrote: post #25

Earlier quoted context omitted.

The R standard library is kind of weird and hard to use for "general purpose" programming. It's very clearly a domain specific language. Also, it's unbelievably slow for basic operations like looping, function calls, and variable assignment. It's literally orders of magnitude slower than Python (I've tested it). Unlike its fellow C-flavored-Lisp Javascript, R retains an extreme level of homoiconicity, Which apparentl…

I can give you some sense of why one can compile SBCL and not R. SBCL and other Lisps these days maintain some kind of conceptual barrier between macroexpansion time and run time, so that you can _stage_ your activity appropriately. Read the code, gather the macro definitions, expand the code (repeat as necessary) until you have some base language without metaprogramming in it. Then you can optimize and compile. This…

> This is particularly true of Scheme which foregoes much of the dynamic quality of Common Lisp

The idea of even the standard Common Lisp is that both is possible: a static Common Lisp and a dynamic Common Lisp, even within the same application in different sections of the program. Common Lisp allows hints to the compiler to remove various features (like fully generic code being reduced to type specific code), it allows a compiler to do various optimizations (inlining, getting rid of late-binding, ...) - while at the same time other parts of the code might be interpreted on the s-expression level.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#100

Post by goatlover » Fri, Jul 05, 2019, 10:08 PM UTC

vincent-toups wrote: post #75

nerdponx wrote: post #25

Earlier quoted context omitted.

The R standard library is kind of weird and hard to use for "general purpose" programming. It's very clearly a domain specific language. Also, it's unbelievably slow for basic operations like looping, function calls, and variable assignment. It's literally orders of magnitude slower than Python (I've tested it). Unlike its fellow C-flavored-Lisp Javascript, R retains an extreme level of homoiconicity, Which apparentl…

I can give you some sense of why one can compile SBCL and not R. SBCL and other Lisps these days maintain some kind of conceptual barrier between macroexpansion time and run time, so that you can _stage_ your activity appropriately. Read the code, gather the macro definitions, expand the code (repeat as necessary) until you have some base language without metaprogramming in it. Then you can optimize and compile. This…

I didn't think the with statement was removed, as that would break backwards compatibility on the web with any older code that used it. Instead, it's just not allowed under strict mode, but it is still part of the ECMAScript specification. I just tried it in the Chrome console, and it still works.