An opinionated view of the Tidyverse “dialect” of the R language • Hacker News Board

Re: An opinionated view of the Tidyverse “dialect” of the R language

#131

Post by kgwgk » Sat, Jul 06, 2019, 11:51 AM UTC

roenxi wrote: post #128

kgwgk wrote: post #126

Earlier quoted context omitted.

One could also say that the second code sample is someone unwinding something that they are conceptualising as a set of nested operations: the_data x), variable_c = variable_a/variable_b), 100) All of them are equivalent, each representation has its advantages and disadvantages.

Well, yeah, if anyone wants to write their pipes that way that is also an option. They aren't going to though, that isn't really human readable when there is a sensible option like using a pipe operator.

If you find that the example above is not "human readable" then the following code will make your head explode because it cannot be transformed into a pipe representation (or at least it's not so straightforward):

  the_data  x),
                         variable_c = variable_a/variable_b)

Re: An opinionated view of the Tidyverse “dialect” of the R language

#132

Post by hadley » Sat, Jul 06, 2019, 11:55 AM UTC

RosanaAnaDana wrote: post #107

hadley wrote: post #101

Earlier quoted context omitted.

You are of course free to dispute what is in the tidyverse, but I pretty strongly believe that it is part of the tidyverse

Seeing as the tidyverse is pretty much your invention, its well within your purvue to define what is in and what is out. I do think, respectfully, that both the op and the general sentiment article have the right of it. Data.table and data.table-esque notation represent such an improvement of tibble/dplyr, that within my company, we're making a concerted effort to purge all tidyverse packages from general use (less g…

If you find data.table more useful, you should by all means use it.

My greatest regret about coining the word tidyverse is that for some reason people seem to think it’s a monolith. It’s not; you’re totally free to pick and choose whatever parts of it you find useful.

It doesn’t hurt my feelings if packages that I have help write aren’t the perfect fit for your problems. Use whatever makes you happy :)

Re: An opinionated view of the Tidyverse “dialect” of the R language

#133

Post by hadley » Sat, Jul 06, 2019, 11:58 AM UTC

Bootvis wrote: post #120

hadley wrote: post #101

Earlier quoted context omitted.

You are of course free to dispute what is in the tidyverse, but I pretty strongly believe that it is part of the tidyverse

Thank you for your reply and all your work. I probably should have phrased my comment better. My point was that it’s perfectly possible to use ggplot without explicit knowledge of tidy principles or tibbles. This is great! And for example, I’m reading through your ggplot book and it doesn’t make reference to it.[1] In my work, we use data.tables (we believe we need the raw performance) with ggplot for visualization a…

Pretty much all of the individual components of the tidyverse were created before the tidyverse since it’s only 3 years old.

But there’s no reason to use only the tidyverse. That’s not something I’ve ever recommended and it would be extremely hard. I just object to people claiming that some of the most important parts of the tidyverse aren’t actually parts of it.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#134

Post by roenxi » Sat, Jul 06, 2019, 12:12 PM UTC

kgwgk wrote: post #131

roenxi wrote: post #128

Earlier quoted context omitted.

Well, yeah, if anyone wants to write their pipes that way that is also an option. They aren't going to though, that isn't really human readable when there is a sensible option like using a pipe operator.

If you find that the example above is not "human readable" then the following code will make your head explode because it cannot be transformed into a pipe representation (or at least it's not so straightforward): the_data x), variable_c = variable_a/variable_b)

Maybe consider:

  the_data %
            rbind(read.csv('/path/to/data/file2.csv')) %>%
            filter(variable_a > x) %>%
            mutate(variable_c = variable_a/variable_b)

You've thrown out a few variations on a theme here, but I'm not sure where you are trying to angle towards. There are a lot of ways to format code, and I can't tell you what works best for you.

But this isn't shaking the paradigm of having an initial block of data (here separated a little awkwardly into two files) that is being subjected to a series of functional transformations, which is sorta pipe-like. Being able to talk about this as a 'pipe' is hugely useful, and having a pipe operator really standardises how people can talk about it. In my experience, selling this as a concept to a beginner is a lot easier when it has syntax support so they know they are doing the 'right thing' and get error messages/uncomfortable looking code when they try to cheat. I don't really care how you nest the brackets, if you want to write something that isn't a pipe go with (edit obviously a non-buggy version of...):

  for i in 1:nrow(df){
   if(df[i,'variable a'] > df[i,'x']) continue;
   df[i,'variable ac'] = df[i,'variable_a']/df[i,'variable_b']
  }

If someone shows me that and I can tell them to 'go rewrite it using pipe operators, it will be better' they will naturally start seeing that that horrible block of mixed ideas can be separated out into a few basic composable operations, and that is funnily enough easier for a beginner to grasp in my experience teaching people to use R. Plus, they can tell that a statement is going to be a series of composes on top of raw data if they see a pipe operator. This is a useful thing that makes it easier to read the code.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#135

Post by Zelazny7 » Sat, Jul 06, 2019, 1:19 PM UTC

ACow_Adonis wrote: post #117

nonbel wrote: post #102

Earlier quoted context omitted.

> "Dr. Matloff is grossly underestimating how painful it is for beginners to, eg, figure out when they are dealing with a column of strings vs a column of factors." This is just sapply(df, class)... Is this not found in R 101 material that beginners are likely to find?

Pay no attention to the myriad of other apply functions hiding behind the curtain and the difficulty in remembering what each of them does and why you can't just use map, eh? :) So to answer your question, no, its not just simply that.

I use a simply device to keep these straight:

- (l)apply: List apply always returns a list

- (s)apply: simplify apply tries to return simplified result

- (v)apply: verify apply checks the return type conforms to user supplied example

- (m)apply: multiple apply applies FUN to multiple vectors

- (r)apply: recursive apply is essentially a flatmap

- apply : no device here, only use on matrices, never data.frames

Re: An opinionated view of the Tidyverse “dialect” of the R language

#136

Post by kkoncevicius » Sat, Jul 06, 2019, 1:21 PM UTC

mespe wrote: post #65

FranzFerdiNaN wrote: post #61

Earlier quoted context omitted.

"Meanwhile, none of the truly advanced R users I encounter use the Tidyverse." This is most likely because truly advanced R users have been using R since far before the Tidyverse existed. A whole new generation of R users is being brought up with the Tidyverse, so im curious to see how the situation will be in 10 years time.

This is not entirely true - many are of the same "generation" as me. To be clear, I don't actually consider myself an advanced user. To me, many of these advanced users are pushing against the limits of the language in a way you don't see with run of the mill data science. That said, I have heard my mentor say "I don't get the point of - I did this 20 years ago."

> That said, I have heard my mentor say "I don't get the point of - I did this 20 years ago."

Tidyverse has a tendency to promote "new" things without any references to what came before. If you listen to their talks they speak as if they invented functional programming and the idea of pure and tiny functions working together.

And it trickles down to the users. A lot of people who learned tidyverse first for example, praise the `purrr` package, but have no idea that something like `Map()` is in base R.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#137

Post by kgwgk » Sat, Jul 06, 2019, 1:32 PM UTC

roenxi wrote: post #134

kgwgk wrote: post #131

Earlier quoted context omitted.

If you find that the example above is not "human readable" then the following code will make your head explode because it cannot be transformed into a pipe representation (or at least it's not so straightforward): the_data x), variable_c = variable_a/variable_b)

Maybe consider: the_data % rbind(read.csv('/path/to/data/file2.csv')) %>% filter(variable_a > x) %>% mutate(variable_c = variable_a/variable_b) You've thrown out a few variations on a theme here, but I'm not sure where you are trying to angle towards. There are a lot of ways to format code, and I can't tell you what works best for you. But this isn't shaking the paradigm of having an initial block of data (here separ…

My point is that sometimes you don't have a single "thread" of calculation. Putting rbind into a pipe like you did is somewhat artificial (broken symmetry) and doesn't work so well if there is some pre-processing before the merge and some post-processing after the merge (or if we have two or more essentially different arguments in a function that need some preprocessing). You may say that having multiple pipes, one merge operation (or whatever the function with multiple inputs is), and then a downstream pipe is the "human readable" way to do that. I'm not sure if that makes programmers able to handle some nesting superhuman or subhuman :-)

Teaching a "paradigm" can be too limiting. Looking at some random tutorial on the web:

"To demonstrate the above advantages of the pipe operator, consider the following example.

  round(cos(exp(sin(log10(sqrt(25))))), 2)
  # -0.33

"The code above looks messy and it is cumbersome to step through all the different functions and also keep track of the brackets when writing the code. "The method below uses magrittr‘s pipe (%>%) and makes the function calls easier to understand.

  sqrt(25) %>%
    log10() %>%
    sin() %>%
    exp() %>%
    cos() %>%
    round(2)
  # -0.33

Really? Do we want to teach people that the code below is so much better than the code above?

What do we expect them to do if they find something like

  100*exp(cumsum(0.6*diff(log(STOCKS))+0.4*diff(log(BONDS))))

which is a perfectly readable way of calculating the evolution of the value of a 60/40 portfolio of stocks and bonds from the value of each component at each rebalancing date?

http://thatdatatho.com/2019/03/13/tutorial-about-magrittrs-p...

Re: An opinionated view of the Tidyverse “dialect” of the R language

#138

Post by hadley » Sat, Jul 06, 2019, 2:58 PM UTC

kgwgk wrote: post #137

roenxi wrote: post #134

Earlier quoted context omitted.

Maybe consider: the_data % rbind(read.csv('/path/to/data/file2.csv')) %>% filter(variable_a > x) %>% mutate(variable_c = variable_a/variable_b) You've thrown out a few variations on a theme here, but I'm not sure where you are trying to angle towards. There are a lot of ways to format code, and I can't tell you what works best for you. But this isn't shaking the paradigm of having an initial block of data (here separ…

My point is that sometimes you don't have a single "thread" of calculation. Putting rbind into a pipe like you did is somewhat artificial (broken symmetry) and doesn't work so well if there is some pre-processing before the merge and some post-processing after the merge (or if we have two or more essentially different arguments in a function that need some preprocessing). You may say that having multiple pipes, one m…

I don’t think anyone is arguing that the pipe should be the -only- form of composition. Just that it’s a useful form when you have a linear sequence of transformations. Sometimes it’s useful to force something close to being linear into a linear form for consistency, but typically you would switch to an alternate form of composition, typically assigning to intermediate variables.

It’s easy to find examples of using the pipe in ways that I would consider suboptimal. But that doesn’t affect my thesis that, on average, the use of the pipe leads to more readable code.

I was also initially quite sceptical of the pipe, since it is a fundamentally new syntax for R (although obviously used in many other languages). I think the uptake by a wide variety of people across the R community does suggest there’s something there.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#139

Post by BoiledCabbage » Sat, Jul 06, 2019, 2:59 PM UTC

Zelazny7 wrote: post #135

ACow_Adonis wrote: post #117

Earlier quoted context omitted.

Pay no attention to the myriad of other apply functions hiding behind the curtain and the difficulty in remembering what each of them does and why you can't just use map, eh? :) So to answer your question, no, its not just simply that.

I use a simply device to keep these straight: - (l)apply: List apply always returns a list - (s)apply: simplify apply tries to return simplified result - (v)apply: verify apply checks the return type conforms to user supplied example - (m)apply: multiple apply applies FUN to multiple vectors - (r)apply: recursive apply is essentially a flatmap - apply : no device here, only use on matrices, never data.frames

Love it or hate it, it's pretty clear these fns had very little api design thought go into them.

As much as I think people can lean too much on the concept, single responsibility principle would've gone a long way here.

Re: An opinionated view of the Tidyverse “dialect” of the R language

#140

Post by roenxi » Sat, Jul 06, 2019, 3:01 PM UTC

Zelazny7 wrote: post #135

ACow_Adonis wrote: post #117

Earlier quoted context omitted.

Pay no attention to the myriad of other apply functions hiding behind the curtain and the difficulty in remembering what each of them does and why you can't just use map, eh? :) So to answer your question, no, its not just simply that.

I use a simply device to keep these straight: - (l)apply: List apply always returns a list - (s)apply: simplify apply tries to return simplified result - (v)apply: verify apply checks the return type conforms to user supplied example - (m)apply: multiple apply applies FUN to multiple vectors - (r)apply: recursive apply is essentially a flatmap - apply : no device here, only use on matrices, never data.frames

After reading your explanation I still, as has been the case for years, don't understand what sapply or rapply does, and vapply sounds weird.

That isn't going to change, because I'm just not going to use them or 'invest' the time in finding out what some statistician-of-yore's interpretation of a map is. Instead I'll stick to tidyverse map - returns a list. Or tidyverse map_[int/chr/dbl/etc, etc] if I want a vector of [int/chr/dbl/etc, etc].

That and the data frame manipulation verbs covers the most useful 80% of cases where *apply would otherwise be needed. If the base R team were implementing functions that way in the base, stats professors wouldn't need to complain about mass exoduses from base R.