91–100 of 124 posts
You can install specific package versions recorded in environment.yml file.
There are probably many ways to do this but this is an approach I like.
https://docs.anaconda.com/anaconda/user-guide/tasks/using-r-...
Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn'…
> You give me your code and enough information for me to produce and identical environment or (even better) your code is insenstive the environment, then your research is Repeatable.
> If you describe your study sufficiently well that I can re-implement your study from scratch, without looking at your code and still get the same answer, then it is Reproducible
> If I can arrive at the same conclusions as you, just from a description of its aims, then it is Replicable.
Reproducible? Or deterministic? There's certainly benefits to being able to pull down research source code, and bug checking it. That's how programmers check code: tests and audits. However I think reproducing research is more often then not done "from scratch", taking a new sample, treating it, checking results. "independent verification". Re-using source code saves time, but I would argue not being able to shouldn'…
That's something that eludes software people: reproducibility in science is the ability to create independent tests. Making software available, while useful, does very little for reproducibility from the scientific point of view.
The coding standards are often abysmally, unexpectedly terrible. Often not even the help of the original authors is enough to be able to produce the same figures from a paper because things and settings and commands get forgotten. Some part of the analysis was done in one language, another part in Excel. Some of the code has now disappeared. Some of the libraries are no longer working. Some people left and their academic storage space was wiped and therefore the intermediate steps and results or notes are deleted. You wouldn't believe it.
Once a paper is published researchers are not really incentivized to document things or maintain the materials. They got the publication, they put it on their CV. On to the next project! No time to waste on work that's already completed. New work leads to new publications, messing around with the old code for the sake of a potential later person interested in it is a waste from the point of view of a researcher, career wise. Also most papers are never attempted to be reproduced ever.
Earlier quoted context omitted.
If I am understanding this correctly, the problem is that the paper authors do not provide a specific version or a package.json equivalent. In that case, using dates seem to be the only choice.
Even if that's the case, using dates isn't a solution because dates don't give you the build that the researcher used. Date of publication is different from the date when the code ran and there is no guarantee that the researcher ran the latest version of every dependency that was available to them anyway. In fact that's very unlikely considering that some their libraries might require older versions. It might not ev…
Earlier quoted context omitted.
Even this isn’t enough to be reproducible for complex numeric code as switching CPU can make a big difference with small differences being amplified. Hopefully none of those cases matter but it’s hard to definitively prove that.
If the research results depend on small differences being amplified you have a much much bigger problem. (but if course this could happen unnoticed/sloppy work)
If you can't even get those numbers, then you can suspect any number of things. Maybe you're not using the right data, maybe there was a typo, maybe someone fraudulently manually tweaked the numbers, maybe you forgot to do a step in the processing chain etc etc. There's no way to know what's going on if you can't even be sure how the original numbers were created.
Either I'm misunderstanding or this is a non-problem. You can specify older versions of a package when you install it. You can also manage them with packrat. As long as researchers share their language and package versions, you can fully reproduce their environment. (And the base language is really stable, almost to a fault.) This is just a bad way for the author to promote their own library for dealing with this. Th…
[1]: https://cran.r-project.org/web/packages/versions/index.html
Earlier quoted context omitted.
Even if that's the case, using dates isn't a solution because dates don't give you the build that the researcher used. Date of publication is different from the date when the code ran and there is no guarantee that the researcher ran the latest version of every dependency that was available to them anyway. In fact that's very unlikely considering that some their libraries might require older versions. It might not ev…
So, what is your better alternative then? I honestly believe using version available at that date is better than using the latest version.
Earlier quoted context omitted.
As I can see from the researchers in our cluster and my own academic research, most people still avoid spaces in paths and files like the plague. YMMV of course.
If my own hobby python projects are anything to go by, there aren’t even folders ;-) I have a friend who taught herself R for her research and it was basically one big procedural codebase.
Sarcasm aside, I've worked with codebases like that- thousand-line java methods and classes and the like. The problem is that there's nothing that really forces modularity on a codebase. There isn't even any consensus, objective way to modularise code. Otherwise, a machine could do it and we wouldn't have this kind of problem. But, a machine cannot, and so we do.