Live data from Hacker News

An opinionated view of the Tidyverse “dialect” of the R language

github.com

81–90 of 156 posts

Re: An opinionated view of the Tidyverse “dialect” of the R language

#81

From personal experience, I learned R during college as a part of my stats courses, before tidyverse even existed. After doing a few personal projects using base R with complex manipulations, I would have quit using R entirely if it were not for tidyverse/dplyr. I'm a pragmatic software engineer: I prefer to get a data analysis job as fast as necessary, and IMO there isn't much merit in making things more complicated…

This also looks painful: https://community.rstudio.com/t/magrittr-inside-a-package/20...

Re: An opinionated view of the Tidyverse “dialect” of the R language

#82
I am a feverent user of data.table and try to avoid dplyr when ever possible. I have colleagues who are exactly the opposite. No one cares.

If you want efficient code, try not to use dplyr. If you want readable code, data.table isn't the best answer. If you want speed, there are better languages than R to choose from, even with Rcpp.

If you want to get something done, go with whatever you know best and will do the job.

Avoid Stata.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#83
post #80
post #52

I teach and consult on R and data science. I had the privilege of learning R from one of R's core developers. My students often ask why I don't use the Tidyverse. The answer is because I don't need to - I can do everything the Tidyverse does and so much more in base R. This article only briefly touches on what I think is the biggest issue with the Tidyverse. The Tidyverse is incredibly limiting. The "tidy" workflow h…

Can you provide an example of a problem that requires multiple tidyverse operations, but could be solved equally well using only lapply?

I run a large scale national survey. We download the data from our survey platform. Survey respondents are asked 100+ questions. The questions change week to week and so the column names are not consistent. We exclude respondents who appear to be cheating the system (rushing through questions, straight-lining, skipping almost every question, etc.) As part of our completion check, we want to do a row-wise map of the data frame e.g. apply(respondents, 1, completion_function). For the sake of this post, let's say that the desired completion function is simply to return the number of NAs in the row -- sum(is.na(x)). I don't care about the name of the variables, I just want to treat each row as a vector one at a time, then perform the operation, and return it as a mutated variable.

When I wrote code to do this, the preferred tidyverse routes would either be to 1:nrow(df) %>% map(function(index) { row = df[index, ] }) or else df %>% pmap(function(.... laundry list of variables here)) { }. A brief Google shows this has gotten worse in the last few years as dplyr deprecated rowwise operations. I do see stack overflow posts of people writing their own tidy/pipe-friendly row-wise iteration functions, but nothing official.

Maybe I missed something. I am a reasonably competent R programmer and package author, but I don't live and breathe tidyverse data-wrangling the way some people do.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#84
post #29

Earlier quoted context omitted.

I think the article sort of punts on providing examples of a complicated set of operations on a data frame. dplyr's author provides what I think is a good example of the differences between data.table and dplyr on a reasonably complex problem: https://stackoverflow.com/questions/21435339/data-table-vs-d... I feel like the first example is far more readable than the second. People can disagree on this, but the adoptio…

The second example should be written like this: diamondsDT[cut != "Fair", .(AvgPrice = mean(price), MedianPrice = as.numeric(median(price)), Count = .N), cut][order(-Count)] There is no need to break it to 10 lines.

I honestly find that less readable than Hadley's version, especially turning "by = cut" into "cut." This is where terseness really cuts into readability (and positional arguments is one of my least favorite features about R in terms of long-term readability of code).

Re: An opinionated view of the Tidyverse “dialect” of the R language

#85

From personal experience, I learned R during college as a part of my stats courses, before tidyverse even existed. After doing a few personal projects using base R with complex manipulations, I would have quit using R entirely if it were not for tidyverse/dplyr. I'm a pragmatic software engineer: I prefer to get a data analysis job as fast as necessary, and IMO there isn't much merit in making things more complicated…

With regards to the usefulness of piping with dplyr: the post does express its admiration for data.table as the alternative for dplyr.

In data.table, chaining operations together is standard and as simple as a set of brackets to contain the next operation.

Both are preferable to pandas though (joke not troll).

Re: An opinionated view of the Tidyverse “dialect” of the R language

#86
post #81

From personal experience, I learned R during college as a part of my stats courses, before tidyverse even existed. After doing a few personal projects using base R with complex manipulations, I would have quit using R entirely if it were not for tidyverse/dplyr. I'm a pragmatic software engineer: I prefer to get a data analysis job as fast as necessary, and IMO there isn't much merit in making things more complicated…

This also looks painful: https://community.rstudio.com/t/magrittr-inside-a-package/20...

That's an interesting caveat with pipes/magrittr (I didn't know of it), although it's out of scope of this debate since this discussion is just about how tidyverse uses pipes.

In my experience if there's a package that's required in a script/R Notebook outside of tidyverse, I just import the entire library at the start of the file Python-style (to make the dependency obvious), which would avoid this issue.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#87
post #81

Earlier quoted context omitted.

This also looks painful: https://community.rstudio.com/t/magrittr-inside-a-package/20...

That's an interesting caveat with pipes/magrittr (I didn't know of it), although it's out of scope of this debate since this discussion is just about how tidyverse uses pipes. In my experience if there's a package that's required in a script/R Notebook outside of tidyverse, I just import the entire library at the start of the file Python-style (to make the dependency obvious), which would avoid this issue.

If you claim that something like

  the_data  x)
  the_data 
is a pain to read compared to

  the_data %
    subset(variable_a > x) %>%
    transform(variable_c = variable_a/variable_b) %>%
    head(100)
I thinks it’s fair to mention how by making things more complicated by using pipes you expose yourself to other issues.

(In my opinion the first variant is not less readable, and it has the advantage of allowing any of the operations to be removed by commenting the corresponding line, while in the second case to remove the last operation you also need to remove the piping operator in the preceding line.)

[Edit: I’m not sure I understand your “out of scope” remark. You don’t mean that magrittr is outside of tidyverse, do you?]

Re: An opinionated view of the Tidyverse “dialect” of the R language

#88
post #43

Earlier quoted context omitted.

That sounds incredible to me as using pandas without falling into the traps of chained indexing seems so much harder than 90% of Python.

I've been planning to go from "can work it out with copious examples" to "knows when to use apply, transform, etc. off the cuff" level of knowledge in Pandas - do you have any suggestions on good resources on this? I'd like to understand better why the data structures work the way they do and thus have an intuition on what operations to use when. The O'Reilly Python Data Science Handbook[0] seems like it might be use…

I find the pandas documentation pretty good [0]. Actually, I find the documentation, official tutorials of pandas, numpy, matplotlib all pretty good. Each package has their own idiosyncrasies but I think the documentation and code examples cover them well enough.

The OReily book should not be that outdated if at all. It also covers other essential tools.

[0] https://pandas.pydata.org/pandas-docs/stable/getting_started...

Re: An opinionated view of the Tidyverse “dialect” of the R language

#89

I was waiting for the critique... but I never quite saw it. Imho, the data problem Tidyverse is trying to solve is basically the ones we face in a database. So, select, join, inner join and so forth. Show me all the rows in this datatable where the 4th columm is larger than the 6th column and the number itself is odd. Something like that. There might be other ways to do it, but you want your select, filter, summarize…

His critique is more about the impact of the full ecosystem effect of the Tidyverse, not what you are referring to, which is just the dplyr semantics. The Tidyverse demands that it's many related packages use tidy data principles and lock users into that approach, which differs from base-R. Much of this discussion is really just a debate about dplyr and magrittr rather than the fragmentation that the broader tidyverse has brought on. All that said, I agree with many commenters that the Tidyverse's improvements to speed of development can more than offset the speed of execution issues, at least for small-to-medium datasets.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#90

I recently tried to do some stuff in R with tidyverse, and was not a fan. I'm no expert, but the tidyverse's frequent use of nonstandard evaluation drove me crazy. It makes it so much more difficult to write functions encapsulating tidy functions. However, I never see people complain about this, so maybe I'm doing something wrong or not grokking something...

It's because RStudio are pushing R/tidyverse as an interactive datascience environment, so most of the tutorials you see will be aimed at users sending a few lines a time to the REPL. Programming with the tidyverse and its non-standard evaluation really is a pain.
Post reply on HN