Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discovering copy-on-write in R

franklin.dyer.me

Post reply on HN
Print view View original

1–10 of 23 posts

Re: Discovering copy-on-write in R

#3

Post by JZL003 »

Copy on write is really nice, especially when I often face a very very large read-only matrix (200+GB) and want to do some embarrassingly parallel processes on subsets of it. I haven't found a language which makes it as easy, not python (although not unexpected), not Julia even

Re: Discovering copy-on-write in R

#4

Post by mplewis9z »

JZL003 wrote: post #3

Copy on write is really nice, especially when I often face a very very large read-only matrix (200+GB) and want to do some embarrassingly parallel processes on subsets of it. I haven't found a language which makes it as easy, not python (although not unexpected), not Julia even

If you haven’t tried Swift, copy-on-write is one of the core tenets of its value types (`struct`s, basically), and it’s almost entirely transparent.

Re: Discovering copy-on-write in R

#6

Post by sampo »

JZL003 wrote: post #3

Copy on write is really nice, especially when I often face a very very large read-only matrix (200+GB) and want to do some embarrassingly parallel processes on subsets of it. I haven't found a language which makes it as easy, not python (although not unexpected), not Julia even

> not python

Pandas has a global option to turn on copy-on-write.

https://pandas.pydata.org/docs/dev/user_guide/copy_on_write....

Re: Discovering copy-on-write in R

#7

Post by 2devnull »

>” How to prevent this? The obvious way is to just not use dataframes, at least not while doing aggregation. Rather than allocating a huge dataframe and loading our partial results into its columns bit by bit, we can just store our partial results in a plain list.”

A lot to be said for not defaulting to data frames, in both r and python. Or, if you must, using something like r data.table or python’s polars if you don’t think in other data structures easily or just want convenience.

Re: Discovering copy-on-write in R

#8

Post by franklin_p_dyer »

2devnull wrote: post #7

>” How to prevent this? The obvious way is to just not use dataframes, at least not while doing aggregation. Rather than allocating a huge dataframe and loading our partial results into its columns bit by bit, we can just store our partial results in a plain list.” A lot to be said for not defaulting to data frames, in both r and python. Or, if you must, using something like r data.table or python’s polars if you don…

For sure! Definitely an good thing to know for an R newbie like me who is handling large datasets naively.

Thanks for mentioning polars, I hadn't heard of it before but it looks neat.

Re: Discovering copy-on-write in R

#9

Post by ryanmonroe »

Good article. Some smaller changes you could make to the final function: In the last line `as.data.frame(do.call(cbind, out_list))` is used to convert the list to a data.frame. Passing it to `cbind` converts the list to a matrix (i.e. combines it into one long vector internally), and then `as.data.frame` converts it back to a list (as noted in the article data frames are lists). Instead, you can use `as.data.frame(out_list)` to make your list a data frame directly, to avoid converting the list to a matrix and back to a list again. The `unlist(lapply(split(cvec, groups), aggfun))` is also doing a lot of work, if you don't mind using an external package*, `collapse::BY(cvec, groups, aggfun)` is much faster (doesn't require converting `groups` to factor, doesn't copy `cvec`'s contents like `split`).

Here's some completely not-the-point of the article code review since I can't help myself. If you can set up earlier steps give you a named list for `col_grouping`, and use `lapply`, the code is a little more concise:

    efficient_flow_agg 
          lapply(collapse::BY, groups, gp$aggfun) |>
          gp$postproc()
      }
      col_grouping |>
        lapply(make_postproc, groups = dat[[gpcol_name]]) |>
        as.data.frame()
    }
* I had previously written here that `tapply` is probably faster, but apparently `tapply` does exactly `unlist(lapply(split(x, g), f)))` anyway? wtf R. Strange there's not something like `collapse::BY` in base R.

Re: Discovering copy-on-write in R

#10

Post by IKantRead »

2devnull wrote: post #7

>” How to prevent this? The obvious way is to just not use dataframes, at least not while doing aggregation. Rather than allocating a huge dataframe and loading our partial results into its columns bit by bit, we can just store our partial results in a plain list.” A lot to be said for not defaulting to data frames, in both r and python. Or, if you must, using something like r data.table or python’s polars if you don…

> A lot to be said for not defaulting to data frames, in both r and python

I would even add especially in Python. The main issue I have found is that pandas heavy code is just not as easy to integrate into other Python tools/features/abstractions as code using mostly numpy, dictionaries and various comprehensions to do the vast majority of your work.

As a heavy pandas user for several years, I decided about a year ago to not import pandas by default and instead treat most data problems like regular python problems. I've been genuinely surprised as how much easier it is to create useful abstractions with the code I've been writing, and also how much easier it's been to onboard non-DS devs into the code base.

There are a few obvious cases when Pandas is very helpful, and I'll pull it out in those places, but I've been able to do a tremendous amount of data work in the last year and used very little pandas. The result is that I have an actual codebase to work with now rather than a billion broken notebooks.

Post reply on HN