Groundhog: Addressing the Threat That R Poses to Reproducible Research
1–10 of 124 posts
Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research
#2I’ve had some brief run-ins with R, and it doesn’t surprise me that it doesn’t have a versioning story for packages, and that the patched-in system described here is based on dates rather than something like a SHA or version number...
My favorite description of the language comes from http://arrgh.tim-smith.us/:
> R is a shockingly dreadful language for an exceptionally useful data analysis environment.
I feel like this is just one more data point to support that statement.
Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research
#3I know of two other existing solutions to this, although I don't know enough to compare. I don't think either of these tick all the author's boxes.
Microsoft MRAN https://mran.microsoft.com/
> For the purpose of reproducibility, MRAN hosts daily snapshots of the CRAN R packages and R releases as far back as Sept. 17, 2014.
MRAN doesn't seem to be very well known or used in the R community, but I don't really know why?
Separately, Nix https://nixos.org/ also solves this problem for lots of different languages, but is difficult to get started with and still a bit rough around the edges. Probably not a good recommendation for a typical analyst or academic at this point.
Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research
#4Nothing about this is specific to R.
If you want to guarantee reproducible results you have to use a container/image with libraries added at build time. Anytime you are relying on floating versions or downloaded libraries you will have issues.
Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research
#5Does R not allow you to lock into versions?
Is this person suggesting we never improve anything? :)
Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research
#6I’ve yet to use it personally, but renv [1] seems to try to solve the reproducible builds problem in a way more similar to other modern package managers (e.g. by generating a lockfile).
This approach enables stricter validations against tampering with the package repositories as a hash of the package can be stored in the lockfile, however it is obviously a bit more complex to use than the groundhog approach.
Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research
#7I'm curious whether this actually solves the problem. I understand how this assists with reproducibility of packages, but the R software itself is updated frequently, as is briefly noted in the preamble to this document. Indeed, the release notes [0] are fairly transparent about the relatively long list of changes.
Given this, it almost seems more dangerous to imply through this package that a particular date's results are reproducible, since unless the user has the same version of R, they may see different results anyway.
[0]: https://stat.ethz.ch/pipermail/r-announce/2020/000653.html
Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research
#8Reproducible? Or deterministic?
There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits.
However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification".
Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility.
Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research
#9Nothing about this is specific to R. If you want to guarantee reproducible results you have to use a container/image with libraries added at build time. Anytime you are relying on floating versions or downloaded libraries you will have issues.
Yeah or even just vendorize your dependencies.
Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research
#10Nothing about this is specific to R. If you want to guarantee reproducible results you have to use a container/image with libraries added at build time. Anytime you are relying on floating versions or downloaded libraries you will have issues.
Even this isn’t enough to be reproducible for complex numeric code as switching CPU can make a big difference with small differences being amplified. Hopefully none of those cases matter but it’s hard to definitively prove that.