Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groundhog: Addressing the Threat That R Poses to Reproducible Research

datacolada.org

Post reply on HN
Print view View original

81–90 of 124 posts

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#81

Post by st1x7 »

ekianjo wrote: post #78
st1x7 wrote: post #33

Either I'm misunderstanding or this is a non-problem. You can specify older versions of a package when you install it. You can also manage them with packrat. As long as researchers share their language and package versions, you can fully reproduce their environment. (And the base language is really stable, almost to a fault.) This is just a bad way for the author to promote their own library for dealing with this. Th…

Packrat is deprecated now. It is recommended to use renv instead.

Thanks, I just checked renv out (I haven't had to work with R in over a year). Renv looks much better than packrat at a first glance.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#82

Post by meztez »

st1x7 wrote: post #33

Either I'm misunderstanding or this is a non-problem. You can specify older versions of a package when you install it. You can also manage them with packrat. As long as researchers share their language and package versions, you can fully reproduce their environment. (And the base language is really stable, almost to a fault.) This is just a bad way for the author to promote their own library for dealing with this. Th…

Fully agreed. Add `sessionInfo()` output as an appendix to your publication. Should not be too hard to rebuild from that.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#83

Post by myWindoonn »

chalst wrote: post #20
GlennS wrote: post #3

I know of two other existing solutions to this, although I don't know enough to compare. I don't think either of these tick all the author's boxes. Microsoft MRAN https://mran.microsoft.com/ > For the purpose of reproducibility, MRAN hosts daily snapshots of the CRAN R packages and R releases as far back as Sept. 17, 2014. MRAN doesn't seem to be very well known or used in the R community, but I don't really know why…

The article discusses MRAN in footnote 5, when arguing against the MRAN-based 'checkpoint' approach. Nixpkg/Nixos is obviously a useful technology for reproducibility, but note that the output of Nix scripts can depend on the time the system was built, the contents of URLs and the system architecture unless care is taken.

This is misleading; empirically, nixpkgs is about 99% [0] reproducible already. We know that the main variance is between language-specific behaviors; Python, Rust, and C all are prone to reproducibility problems.

In general, we want the output to depend on the system architecture and the contents of URLs. Nix uses hashes to require that URL contents don't change over time, which protects from those contents changing arbitrarily.

[0] https://r13y.com/

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#85

Post by chalst »

myWindoonn wrote: post #83
chalst wrote: post #20

Earlier quoted context omitted.

The article discusses MRAN in footnote 5, when arguing against the MRAN-based 'checkpoint' approach. Nixpkg/Nixos is obviously a useful technology for reproducibility, but note that the output of Nix scripts can depend on the time the system was built, the contents of URLs and the system architecture unless care is taken.

This is misleading; empirically, nixpkgs is about 99% [0] reproducible already. We know that the main variance is between language-specific behaviors; Python, Rust, and C all are prone to reproducibility problems. In general, we want the output to depend on the system architecture and the contents of URLs. Nix uses hashes to require that URL contents don't change over time, which protects from those contents changing…

The current community around NixOS and Nixpkgs handles these issues just fine, but if 'just use Nix' was regarded as a magic bullet for reproducibility in science, I'm guessing it wouldn't work out so well.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#87

Post by hermitcrab »

Being able to assemble a solution from parts (as in R packages) is super flexible. But complex and potentially brittle.

Reproducability is a big problem all around. When I create releases I put the binaries as well as the source in version control, because changes in tools/libraries etc mean that I probably won't be able to create the exact same binary several years later from the same source.

There is always a tradeoff between flexibility and simplicity. Clearly software needs to be able to change, or you are never going to be able to improve it or fix bugs. And an assembly of constantly changing parts is clearly going to come with its own challenges.

My own software product, Easy Data Transform (which competes with R to some extent) trades off some flexibility for simplicity by having a single set of binaries for each platform. You can't add any components (without hacking). So the same version of software should always give the same result.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#88

Post by Hasnep »

kwertzzz wrote: post #49
st1x7 wrote: post #33

Either I'm misunderstanding or this is a non-problem. You can specify older versions of a package when you install it. You can also manage them with packrat. As long as researchers share their language and package versions, you can fully reproduce their environment. (And the base language is really stable, almost to a fault.) This is just a bad way for the author to promote their own library for dealing with this. Th…

I full agree. With version number you can have a good sense whether a package update breaks your code or not (as long the package authors following semantic versioning). I think in julia this problem is solved quite nicely with the Project.toml (list of package that you directly dependent) and Manifest.toml file (the version numbers of the complete dependency tree which is automatically generated). It seems that in g…

Using renv (or packrat) generates a renv.lock file with the version of every dependency.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#90

Post by myWindoonn »

chalst wrote: post #85
myWindoonn wrote: post #83

Earlier quoted context omitted.

This is misleading; empirically, nixpkgs is about 99% [0] reproducible already. We know that the main variance is between language-specific behaviors; Python, Rust, and C all are prone to reproducibility problems. In general, we want the output to depend on the system architecture and the contents of URLs. Nix uses hashes to require that URL contents don't change over time, which protects from those contents changing…

The current community around NixOS and Nixpkgs handles these issues just fine, but if 'just use Nix' was regarded as a magic bullet for reproducibility in science, I'm guessing it wouldn't work out so well.

Fortunately, "just use Nix" doesn't do much on its own. People usually want GCC or another complete C toolchain, a C standard library, etc. and this implies that they will use nixpkgs or one of its forks. If people try to "just use Nix" in anger, then they will almost certainly be funneled into using nixpkgs as a matter of practice.

The main problem with reproducibility in science is that most scientists are not actually interested in doing science. Of course software will not fix this problem.

Post reply on HN