Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groundhog: Addressing the Threat That R Poses to Reproducible Research

datacolada.org

Post reply on HN
Print view View original

21–30 of 124 posts

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#21

Post by bsza »

cauthon wrote: post #15

This title is an exceedingly hot take for someone who wrote a new package manager. Also, it appears that Groundhog is itself a CRAN package and the author recommends installing with install.packages(). So is the author committing to never making any backwards incompatible updates to their new package?

I think it's more like a Wayback Machine for R programs, since the author of a science paper isn't required to use groundhog. You can just provide it the date the article was published, which you already know, and it reconstructs how the program worked on that day.

Also, because groundhog isn't made for the author to use, whether or not the interface changes is irrelevant. You'll never encounter library(groundhog) in a paper.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#22

Post by warlog »

GlennS wrote: post #3

I know of two other existing solutions to this, although I don't know enough to compare. I don't think either of these tick all the author's boxes. Microsoft MRAN https://mran.microsoft.com/ > For the purpose of reproducibility, MRAN hosts daily snapshots of the CRAN R packages and R releases as far back as Sept. 17, 2014. MRAN doesn't seem to be very well known or used in the R community, but I don't really know why…

It looks like this is much more fine grained compared to mran, i.e., with groundhog, you select the date vs with mran where you use the last (often > year old) snapshot.

mran is a great idea and if Rstudio (the defacto gate-keepers of the faith -- with Hadley the high priest) pushed to use mran, then the R community would follow suit (like they do for everything else).

This would do a lot to bring MS into the fold, which would actually be great for R.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#23

Post by asdff »

davnn wrote: post #11
asperous wrote: post #8

Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn'…

> Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility. More often than not it‘s not clear from a paper what exactly the authors did to a achieve a specific result. Being able to exactly reproduce what previous authors did should improve reproducibility; also for new samples.

It's also standard fare for typical lab work. A good paper's methods section would contain enough detail for you to go into your lab and repeat the experiment yourself, even down to the catalog number for the reagents to order from the lab supplier. Code should be no different, that's why it's encouraged that authors submit all code used in analysis and generation of figures.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#24

Post by petters »

coliveira wrote: post #13
asperous wrote: post #8

Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn'…

That's something that eludes software people: reproducibility in science is the ability to create independent tests. Making software available, while useful, does very little for reproducibility from the scientific point of view.

It's true that "reproduce" can mean different things in software and science. But having the code available to "reproduce" any plots in a paper should be a requirement for publication, imo. It certainly is the case for my papers.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#25

Post by jcheng »

qrohlf wrote: post #2

I’ve had some brief run-ins with R, and it doesn’t surprise me that it doesn’t have a versioning story for packages, and that the patched-in system described here is based on dates rather than something like a SHA or version number... My favorite description of the language comes from http://arrgh.tim-smith.us/ : > R is a shockingly dreadful language for an exceptionally useful data analysis environment. I feel like…

Packrat and its successor renv are the most popular package management systems for R, and they are based on versions/SHAs and lockfiles, like most other languages today.

https://rstudio.github.io/packrat/

https://rstudio.github.io/renv/articles/renv.html

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#26

Post by f6v »

coliveira wrote: post #13
asperous wrote: post #8

Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn'…

That's something that eludes software people: reproducibility in science is the ability to create independent tests. Making software available, while useful, does very little for reproducibility from the scientific point of view.

There’re fields of science, like computational biology, where it’s all about the code. I wish the methods section was always 100% unambiguous, but it’s not the case. And nowadays the computational pipelines have to support analysis of up to terabytes of data. You can imagine how many dependencies such pipelines have. Sometimes I have trouble installing the software even when package manager such as anaconda is used.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#27

Post by stewbrew »

I wish the (default) utils::install.packages function could take a version number of the requested library. I also wish library() would automatically install libraries not available on the system. (Both can be achieved with custom functions that shadow the default ones but I would like to see this functionality in the base packages.) Other than that, I think all alternatives to this "threat called R" are worse. It's telling the author has to cite a bug from 2016 for an example of a breaking change.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#28

Post by f6v »

The only working solution I’ve seen is using Docker container with Jupyter Lab and all the dependencies installed. I hate pulling those huge images on my 256GB MBP, but it works. Of course, only bigger labs do that, since individual researchers are often unfamiliar with Docker.

However, if I run my software on HPC cluster, that’s no longer an option. The HPC at my university doesn’t allow running Docker, only Singularity containers(which isn’t supported on Mac).

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#29

Post by Hansi »

warlog wrote: post #22
GlennS wrote: post #3

I know of two other existing solutions to this, although I don't know enough to compare. I don't think either of these tick all the author's boxes. Microsoft MRAN https://mran.microsoft.com/ > For the purpose of reproducibility, MRAN hosts daily snapshots of the CRAN R packages and R releases as far back as Sept. 17, 2014. MRAN doesn't seem to be very well known or used in the R community, but I don't really know why…

It looks like this is much more fine grained compared to mran, i.e., with groundhog, you select the date vs with mran where you use the last (often > year old) snapshot. mran is a great idea and if Rstudio (the defacto gate-keepers of the faith -- with Hadley the high priest) pushed to use mran, then the R community would follow suit (like they do for everything else). This would do a lot to bring MS into the fold, w…

Hadley works for RStudio, RStudio now have their own MRAN type mirror: https://packagemanager.rstudio.com/client/#/

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#30

Post by vhhn »

There are two camps in the R world - tidyverse and base-R (tiniverse).

Its not a coincidence that the author gives an example from the tidyverse ecosystem. Authors and users of tidyverse value other things like consistency and new features over API stability and backward compatility. The base-R ecosystem is actually very stable and so the original package manager is very simple.

With R spreading out from the academic environment and with many new authors breaking their packages' APIs we observe new attempts to solve the issues with dependencies (such as renv or https://rsuite.io)

Post reply on HN