Groundhog: Addressing the Threat That R Poses to Reproducible Research • Hacker News Board

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#31

Post by kgwgk » Thu, Jan 07, 2021, 8:22 AM UTC

warlog wrote: post #22

GlennS wrote: post #3

I know of two other existing solutions to this, although I don't know enough to compare. I don't think either of these tick all the author's boxes. Microsoft MRAN https://mran.microsoft.com/ > For the purpose of reproducibility, MRAN hosts daily snapshots of the CRAN R packages and R releases as far back as Sept. 17, 2014. MRAN doesn't seem to be very well known or used in the R community, but I don't really know why…

It looks like this is much more fine grained compared to mran, i.e., with groundhog, you select the date vs with mran where you use the last (often > year old) snapshot. mran is a great idea and if Rstudio (the defacto gate-keepers of the faith -- with Hadley the high priest) pushed to use mran, then the R community would follow suit (like they do for everything else). This would do a lot to bring MS into the fold, w…

They have their own package management library

https://rstudio.github.io/packrat/

and sell their own package management product

https://rstudio.com/products/package-manager/

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#32

Post by geomark » Thu, Jan 07, 2021, 8:22 AM UTC

asperous wrote: post #8

Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn'…

Back when I studied this stuff there was a distinction between reproducible (rerun analysis on the data from the original experiment and see if you get the same results - if not then there is an error in the analysis) and replicable (redo the entire experiment by taking new data and running the analisys).

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#33

Post by st1x7 » Thu, Jan 07, 2021, 8:33 AM UTC

Either I'm misunderstanding or this is a non-problem. You can specify older versions of a package when you install it. You can also manage them with packrat. As long as researchers share their language and package versions, you can fully reproduce their environment. (And the base language is really stable, almost to a fault.)

This is just a bad way for the author to promote their own library for dealing with this. The way their library seems to approach this (using dates instead of versions) seems horrible too - on any given date I can have a random selection of packages in my environment, some of them up-to-date, some of them not. So unless all researchers start using the author's library (and update to the latest versions of everything just before they publish), it's only making things worse and not really solving the problem it claims to solve.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#34

Post by jsmith99 » Thu, Jan 07, 2021, 8:39 AM UTC

warlog wrote: post #22

GlennS wrote: post #3

I know of two other existing solutions to this, although I don't know enough to compare. I don't think either of these tick all the author's boxes. Microsoft MRAN https://mran.microsoft.com/ > For the purpose of reproducibility, MRAN hosts daily snapshots of the CRAN R packages and R releases as far back as Sept. 17, 2014. MRAN doesn't seem to be very well known or used in the R community, but I don't really know why…

It looks like this is much more fine grained compared to mran, i.e., with groundhog, you select the date vs with mran where you use the last (often > year old) snapshot. mran is a great idea and if Rstudio (the defacto gate-keepers of the faith -- with Hadley the high priest) pushed to use mran, then the R community would follow suit (like they do for everything else). This would do a lot to bring MS into the fold, w…

MRAN takes daily snapshots, and is the repository powering this new package.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#35

Post by jsmith99 » Thu, Jan 07, 2021, 8:44 AM UTC

cauthon wrote: post #15

This title is an exceedingly hot take for someone who wrote a new package manager. Also, it appears that Groundhog is itself a CRAN package and the author recommends installing with install.packages(). So is the author committing to never making any backwards incompatible updates to their new package?

That's the problem. This package is very similar to Microsoft's checkpoint package which is based on Microsoft's MRAN snapshots, and this package also uses MRAN. The article explains the difference is that this package allows you to specify the date in the code itself, whereas checkpoint is used to set a whole installation to a specific date. But this is no advantage as it means code will stop working if the groundhog package changes, whereas with checkpoint a paper could just say 'use packages as of date x'.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#36

Post by st1x7 » Thu, Jan 07, 2021, 8:45 AM UTC

bsza wrote: post #21

cauthon wrote: post #15

This title is an exceedingly hot take for someone who wrote a new package manager. Also, it appears that Groundhog is itself a CRAN package and the author recommends installing with install.packages(). So is the author committing to never making any backwards incompatible updates to their new package?

I think it's more like a Wayback Machine for R programs, since the author of a science paper isn't required to use groundhog. You can just provide it the date the article was published, which you already know, and it reconstructs how the program worked on that day. Also, because groundhog isn't made for the author to use, whether or not the interface changes is irrelevant. You'll never encounter library(groundhog) in…

> and it reconstructs how the program worked on that day.

It reconstructs how the fully updated version of everything worked that day which isn't necessarily the same as the researcher's environment. It's a horrible idea to use dates instead of package versions for this. The author's library doesn't solve the problem it claims to solve.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#37

Post by davnn » Thu, Jan 07, 2021, 8:46 AM UTC

asdff wrote: post #23

davnn wrote: post #11

Earlier quoted context omitted.

> Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility. More often than not it‘s not clear from a paper what exactly the authors did to a achieve a specific result. Being able to exactly reproduce what previous authors did should improve reproducibility; also for new samples.

It's also standard fare for typical lab work. A good paper's methods section would contain enough detail for you to go into your lab and repeat the experiment yourself, even down to the catalog number for the reagents to order from the lab supplier. Code should be no different, that's why it's encouraged that authors submit all code used in analysis and generation of figures.

The fun thing is that there are approaches that want to go beyond this kind of methodological description of a scientific process to code [0, 1]. In general I would say that the more we can remove the human aspect and inherent ambiguity of science, the better for reproducibility. See [2] for a couple of examples.

[0] https://www.emeraldcloudlab.com/ [1] https://nextjournal.com/ [2] https://www.youtube.com/watch?v=L1UgdoP2aeg

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#38

Post by roel_v » Thu, Jan 07, 2021, 8:52 AM UTC

Apart from all the other considerations and problems with various types of package management, consider this:

"Update January 6th, 2021 A reader alerted me to a bug with the current groundhog (version 1.1.0) where you cannot set the groundhog library to be a folder containing spaces in the name."

So we are talking about software here that somehow made it to version 1.1 *without anyone ever using a directory with spaces in it with it". This can be interpreted in two ways: either very few people have spaces in their paths, or very few people have actually ever even tried (not even really used, I'm only talking about the most basic trial use) this package. I'm not a betting man, but if I were, I know where I'd put my money...

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#39

Post by [deleted] » Thu, Jan 07, 2021, 8:55 AM UTC

roel_v wrote: post #38

Apart from all the other considerations and problems with various types of package management, consider this: "Update January 6th, 2021 A reader alerted me to a bug with the current groundhog (version 1.1.0) where you cannot set the groundhog library to be a folder containing spaces in the name." So we are talking about software here that somehow made it to version 1.1 *without anyone ever using a directory with spac…

[deleted]

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#40

Post by bayindirh » Thu, Jan 07, 2021, 9:11 AM UTC

roel_v wrote: post #38

Apart from all the other considerations and problems with various types of package management, consider this: "Update January 6th, 2021 A reader alerted me to a bug with the current groundhog (version 1.1.0) where you cannot set the groundhog library to be a folder containing spaces in the name." So we are talking about software here that somehow made it to version 1.1 *without anyone ever using a directory with spac…

As I can see from the researchers in our cluster and my own academic research, most people still avoid spaces in paths and files like the plague.

YMMV of course.