Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Groundhog: Addressing the Threat That R Poses to Reproducible Research

datacolada.org

Post reply on HN
Print view View original

11–20 of 124 posts

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#11

Post by davnn »

asperous wrote: post #8

Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn'…

> Re-using source code saves time, but I would argue not being able to shouldn't threaten reproducibility.

More often than not it‘s not clear from a paper what exactly the authors did to a achieve a specific result. Being able to exactly reproduce what previous authors did should improve reproducibility; also for new samples.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#12

Post by andi999 »

tempay wrote: post #10
threeseed wrote: post #4

Nothing about this is specific to R. If you want to guarantee reproducible results you have to use a container/image with libraries added at build time. Anytime you are relying on floating versions or downloaded libraries you will have issues.

Even this isn’t enough to be reproducible for complex numeric code as switching CPU can make a big difference with small differences being amplified. Hopefully none of those cases matter but it’s hard to definitively prove that.

If the research results depend on small differences being amplified you have a much much bigger problem. (but if course this could happen unnoticed/sloppy work)

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#13

Post by coliveira »

asperous wrote: post #8

Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn'…

That's something that eludes software people: reproducibility in science is the ability to create independent tests. Making software available, while useful, does very little for reproducibility from the scientific point of view.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#14

Post by jdale27 »

asperous wrote: post #8

Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn'…

Ideally research does get reproduced from scratch; I think what people usually mean when they talk about the replication/reproducibility crisis in science is not being able to reproduce an experiment with new samples, independent data analysis, etc.

However, if you can't even reproduce an analysis with the authors' own data and code, that's a red flag before you even get to the starting line. Ensuring that level of reproducibility is, I think, an essential ingredient to enabling the stronger form of reproducibility.

Personally, I made the mistake during my graduate career of trying to reimplement an analysis using a certain rather complicated ML algorithm, from scratch, in a different language than the original authors had used. After struggling mightily to get it to work, I finally bothered to try to get their own code working. (I had been hesitant to do so because I wasn't proficient in the language they used, and it wasn't even clear they had released all the necessary code, aside from the core algorithm.) Once I did that, I discovered that I couldn't even get their own code working on their own data, and gave up. This was researched published in Science by a group from a top-tier research university. (I don't fully blame the authors, it may well have been my own incompetence that was the issue. But it just serves as yet another illustration of how pervasive and disregarded the reproducibility issue was for a long while.)

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#15

Post by cauthon »

This title is an exceedingly hot take for someone who wrote a new package manager.

Also, it appears that Groundhog is itself a CRAN package and the author recommends installing with install.packages(). So is the author committing to never making any backwards incompatible updates to their new package?

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#17

Post by notafraudster »

anderscarling wrote: post #6

I’ve yet to use it personally, but renv [1] seems to try to solve the reproducible builds problem in a way more similar to other modern package managers (e.g. by generating a lockfile). This approach enables stricter validations against tampering with the package repositories as a hash of the package can be stored in the lockfile, however it is obviously a bit more complex to use than the groundhog approach. [1]: htt…

Agreed that renv is a better solution here. Even the example code for Groundhog is not written in idiomatic R which does not inspire confidence. Simonssohn is a legend in transparent research but not primarily a coder or software tool contributor (take a look at the source for p-curve if you want to see what I mean) and I think a secondary threat to reproducibility is relying on tools that end up abandoned or deprecated or for which bugs never get fixed.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#18

Post by resonantjacket5 »

cauthon wrote: post #15

This title is an exceedingly hot take for someone who wrote a new package manager. Also, it appears that Groundhog is itself a CRAN package and the author recommends installing with install.packages(). So is the author committing to never making any backwards incompatible updates to their new package?

Your take seems a bit 'hot' too?

How else would you install the cran packages without using install.packages? Unless if you want them to recursively install it using groundhog but that seems unnecessary.

As long as you have the timestamp it should work, though I assume there will be some edge case.

What you're saying is like don't use pip because you don't install it using pip? Or don't use package-lock.json because you can't install npm through npm?

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#19

Post by coolreader18 »

cauthon wrote: post #15

This title is an exceedingly hot take for someone who wrote a new package manager. Also, it appears that Groundhog is itself a CRAN package and the author recommends installing with install.packages(). So is the author committing to never making any backwards incompatible updates to their new package?

> So is the author committing to never making any backwards incompatible updates to their new package?

Well, yes, probably. It's not all that hard, and groundhog seems to have a fairly simple API anyways.

And groundhog still uses CRAN packages, it just brings a method of pinning them to a specific version.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#20

Post by chalst »

GlennS wrote: post #3

I know of two other existing solutions to this, although I don't know enough to compare. I don't think either of these tick all the author's boxes. Microsoft MRAN https://mran.microsoft.com/ > For the purpose of reproducibility, MRAN hosts daily snapshots of the CRAN R packages and R releases as far back as Sept. 17, 2014. MRAN doesn't seem to be very well known or used in the R community, but I don't really know why…

The article discusses MRAN in footnote 5, when arguing against the MRAN-based 'checkpoint' approach.

Nixpkg/Nixos is obviously a useful technology for reproducibility, but note that the output of Nix scripts can depend on the time the system was built, the contents of URLs and the system architecture unless care is taken.

Post reply on HN