301–310 of 487 posts
Does code from ten years ago run ever? Try running something on that runs on Python 2 on the current python interpreter today.
Python is an extremely bad example. Try twenty years old Common Lisp code. Or Fortran.
Earlier quoted context omitted.
Guix is one of several solutions that has been touted as a solution. Another one that is quite popular in HPC circles is Spack ( https://spack.readthedocs.io/en/latest/ ). At my institute, we actually tried out Spack for a little bit, but consistently felt like it was implemented more as a research project rather than something that was production-level and maintainable. In large part, this was due to the dependency…
> Spack/Guix/nix are the best solution in theory, but they come with a host of other problems that made them less desirable. I would be quite interested to learn more what these problems are, in your experience. I've only tried Guix (on top of Debian and Arch) and while it is definitively more resource-hungry (especially in terms of disk space), I don't percive it as impractical.
Basically, the learning curve is nearly vertical.
Earlier quoted context omitted.
Data is much trickier because your data source for medical, education or even just regular businesses don't want the added legal weight of making data freely available. This is obviously a shame, I was working on segmentation of open wounds and most papers include a "we are currently in talks with the hospital to make the data available". If you contact the authors directly they will tell you that their committee blo…
It seems like there can be a balance between "the results are unverifiable because no one else can touch the data" and "effectively open-source the dataset"? Something like: "To make it easier to verify the code behind this paper, we've used to generate a synthetic dataset with the same fields as the original and included it with the source code. The isn't comfortable with publishing the full dataset, but they did ag…
This would be administrative overhead, it will be shut down 9 times out of 10. I understand why this might seem easy but it really is not, you can have multiple hospital that each have their committee that agreed to give the researcher their data. They don't have a central authority that you can appeal to, much less someone that can green light your specific access.
As for the synthetic datasets that's basically just having tests and was advocated for elsewhere in this thread.
Earlier quoted context omitted.
As an ex-scientist who used to run lots of simulations, I really fail to see a truly compelling reason why most numerical results (for publication purposes) truly need to publish (and support) deterministic seeding. We've certainly done a lot, scientifically speaking (in terms of post-validated studies), without that level of reproducibility.
If nothing else, it helps debugging code which tries to reproduce your findings.
Earlier quoted context omitted.
I am interested to know the distinction between "production-ready" and "science-ready" code. I do not think "non-experts" should be able to use your code, but I do think an expert who was not involved in writing it should be.
Hard-coded file paths for input data. File paths hard-coded to use somebody's Google Drive so that it only runs if you know their password. Passwords hard-coded to get around the above problem. In-code selection statements like `if( True ) {...}`, where you have no idea what is being selected or why. Code that only runs in the particular workspace image that contains some function that was hacked out to make things w…
As an admin it was quite frustrating, but I understand it sometimes when you know the person/project isn’t tested in a distributed environment. But when it’s the projects that do know how they’re used and still do those things...
Earlier quoted context omitted.
> or in computer science / applied math/stats / etc., with different codebases, with different model variants, on different datasets) and the overall conclusions hold A lot of open sourced CS research is not reproducible. "the code still runs and gives the same output" is not the same as reproducibility.
> A lot of open sourced CS research is not reproducible. I'm not sure if this was meant to be a counter-argument to me, but I completely agree! > "the code still runs and gives the same output" is not the same as reproducibility. Yes, bit-for-bit identical results are neither necessary nor sufficient for reproducibility in the usual scientific sense.
It wasn't :)
Earlier quoted context omitted.
There's a ton of overlap, because science code might be a long running, multi-engineer distributed system and production code might be a script that supports a temporary business process. But let's assume production ready is a multi customer application and science ready is computations to reproduce results in a paper. Here's a quick pass, I'm sure I'm missing stuff, but I've needed to code review a lot of science an…
Your requirements seem to push 'Science ready' far into what I'd consider "worthless waste of time", coming from the perspective of code that's used for data analysis for a particular paper. The key aspect of that code is that it's going to be run once or twice, ever, and it's only ever going to be run on a particular known set of input data. It's a tool (though complex) that we used (once) to get from A to B. It doe…
Earlier quoted context omitted.
This is the same as any other argument against testing. Unless you are actually selling a library, code is not the product. Customers are buying results, not your code base. Yet, we've discovered the importance of testing to make sure customers get the right results without issues. If you want your results to be usable by others, the quality of the code matters. If all you care is publishing a paper, then I guess sur…
But the results are usable by others, in most fields of science the code is not part of these results and is not needed to enjoy, use and build upon the research results. The only case where the code would be used (which is a valid reason why it should be available somehow ) is to assert that your particular results are flawed or fraudulent; otherwise the quality of the code (or its availability, or even existence -…
Not true. Code is often used and reused to churn out a lot more results than the initial paper. A flaw in the code doesn't just show one paper/result as problematic. It can show a large chunk of a researcher's work in his area of expertise to be problematic.
Earlier quoted context omitted.
That's not how the game is played. If you cannot the release the code because the code is too ugly or untested or has bugs, how do you expect anyone with the right expertise to assess your findings? It reminds me of Kerckhoffs's principle in cryptography, which states: A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.
The findings really should be independent of the code. Reproduction should occur by taking the methodology and re-implementing the software and running new experiments.
I myself had very bad experience with extending the undocumented Fortran 77 code (lots of gotos and common blocks) of my supervisor. Finally, I decided to rewrite the whole thing including my new results instead of just somehow embedding my results into the old code for two reasons: (1) I'm presumably faster in rewriting the whole thing including my new research rather than struggling with the old code and (2) I simply would not trust in the numerical results/phenomenology produced by the code. After all, I'm wasting 2 months of my PhD for the marriage of my own results with known results which -in principle- could have been done within one day if the code base would allow for it.
So yes, If it's a one-man-show I would not give too much on code quality (though unit tests and git can safe quite a lot of time during development) but if there is a chance that someone else is going to touch the code in near future it will save time to your colleagues and improve the overall (scientific) productivity.
PS: quite excited about my first post here