Groundhog: Addressing the Threat That R Poses to Reproducible Research • Hacker News Board

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#121

Post by benibela » Thu, Jan 07, 2021, 10:43 PM UTC

Sebb767 wrote: post #116

IshKebab wrote: post #79

Earlier quoted context omitted.

> So we are talking about software here that somehow made it to version 1.1 without anyone ever using a directory with spaces in it with it . This is extremely common, especially on Linux. Basically anything that uses things like Bash or CMake will almost certainly not work in directories containing spaces. Developers don't use paths containing spaces because it causes so many issues with badly written Bash scripts,…

> Developers don't use paths containing spaces because it causes so many issues with badly written Bash scripts, and as a result they don't test their code with paths containing spaces. It is also a PITA to use when typing in a shell, as you need two characters ( \ + space ) instead of one. So even though my scripts can handle them, I still avoid them if possible.

Some programs also use URLs

Today I wanted to send a screenshot by mail.

Should be simple, but with not Gnome. I make the screenshot, Gnome creates a file "Screenshot from ...", but does not tell you where. Then I search it in the file explorer, find it, copy the path. Then I paste the path in the mail program, file:///....Screenshot%20from%20. Then the mail program: "File not found"

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#122

Post by coliveira » Thu, Jan 07, 2021, 10:58 PM UTC

bonoboTP wrote: post #118

coliveira wrote: post #111

Earlier quoted context omitted.

The researcher's job is to properly do an experiment and document it in a paper. If we require more than this, then we will incur in damage to the scientific process for two reasons: (1) companies are not willing to make available software developed by their researchers, therefore they will publish even less; and (2) universities don't have money and staff to produce and maintain software at these standards, so profe…

"These standards" are pretty low. Currently it's a free-for-all chaos. Theoretically papers are reproducible from the documentation found in the paper but that is a lie. It is never reproducible just from the paper. Lots of stuff is done in the background that is not known to the reader. For all we know, they can even tweak their numbers to be 2% better and if someone can't get the results of the paper from the relea…

> "These standards" are pretty low. Currently it's a free-for-all chaos.

I disagree. It is not perfect, but it is certainly a process that enables scientific development, as it has for centuries. If we start to create more and more rules that researchers need to follow, it will become even harder to make scientific research and most institutions won't have resources to continue.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#123

Post by bonoboTP » Thu, Jan 07, 2021, 11:23 PM UTC

coliveira wrote: post #122

bonoboTP wrote: post #118

Earlier quoted context omitted.

"These standards" are pretty low. Currently it's a free-for-all chaos. Theoretically papers are reproducible from the documentation found in the paper but that is a lie. It is never reproducible just from the paper. Lots of stuff is done in the background that is not known to the reader. For all we know, they can even tweak their numbers to be 2% better and if someone can't get the results of the paper from the relea…

> "These standards" are pretty low. Currently it's a free-for-all chaos. I disagree. It is not perfect, but it is certainly a process that enables scientific development, as it has for centuries. If we start to create more and more rules that researchers need to follow, it will become even harder to make scientific research and most institutions won't have resources to continue.

Maybe not rules but expectations culturally among researchers. If you want your work to actually get used, make it usable. I guess it's different in different fields. I'm most familiar with CS and machine learning. There, it's more and more a community expectation to have access to the code or be very skeptical of the numerical results. It doesn't mean the other parts of the paper are also discounted, so if there are genius ideas in the explanation, it is still valuable. But people can do any number of things to squeeze out a benchmark advantage of 1% and I only trust that if I have code (sure, GPU ML code is not fully bitwise reproducible as of now, but it's being worked on. It's becoming technically possible but not everyone has heard about it or understands why). Even without bitwise repeatability, I want the code and instructions how to run the published experiments. I don't believe benchmark claims otherwise. Simply too much noise is being pumped into the literature. To much data torturing for career reasons, for visa reasons, for reasons of "but I must publish to not get fired and finish my PhD, and the reviewers will only let me publish if I beat the benchmark, so I'll do my analysis over and over until I get 1% better".

The culture needs to improve. Benchmarks shouldn't be everything but reviewers are inexperienced. Many reasons in many parts of the system. In other fields the issues are different. They are more about having to obtain statistically significant results or you fail your career.

It's a good thing that people are waking up to this. It's not about punishing the individual researchers, it's about our collective intellectual immune system. We can't digest this firehouse of papers if it's poisoned to such an extent. It's not about charity or burden. It's about being skeptical when we know we're dealing with unreliable data. Science is a massive endeavor with massive quality differences between works and researchers and groups. Blind trust is no longer enough if you care about keeping your beliefs curated.

Re: Groundhog: Addressing the Threat That R Poses to Reproducible Research

#124

Post by ekianjo » Fri, Jan 08, 2021, 12:56 PM UTC

st1x7 wrote: post #81

ekianjo wrote: post #78

Earlier quoted context omitted.

Packrat is deprecated now. It is recommended to use renv instead.

Thanks, I just checked renv out (I haven't had to work with R in over a year). Renv looks much better than packrat at a first glance.

Look at recent Rstudio videos on youtube, they explained why they moved away from packrat.