Live data from Hacker News

Ask HN: Should I publish my research code?

news.ycombinator.com

181–190 of 363 posts

Re: Ask HN: Should I publish my research code?

#181

What a great question. You've come to the right community. My main concern would be to make sure there are no passwords or secret keys in the data, not how it looks. You'll open yourself up for comments. They may be positive or negative. You'll only know how it pans out afterwards. Is the code something that you'll want to improve on for further research? If so publish it on github. It opens the way for others to con…

> My main concern would be to make sure there are no passwords or secret keys in the data, not how it looks. Worth mentioning specifically: If you make a git (et al) repository public, make sure there are no passwords or secret keys in the history of the repository either. Cleaning a repository history can be tricky, so if this is an issue, best to just publish a snapshot of the latest code (or make 100% sure you've…

The brute force way around this is to remove the .git folder and re init the git repo.

For my 2 cents I'd prefer to see sloppy code vs no code.

If you did something wrong, you did it wrong. Hopefully someone would put in a PR to fix it

Re: Ask HN: Should I publish my research code?

#182
Hi, I’m a Research Software Engineer (a person who makes software that helps academics/researchers) at a university in the UK. My recommendation is that not only do you publish the code, you mint a DOI (digital object identifier, Zenodo is usually the go to place for that) for the specific version that was used in your paper and you associate them. And you include a citation file (GitHub supports them now: https://docs.github.com/en/repositories/managing-your-reposi...) in your software repo.

Benefits: people who want to reproduce your analysis can use exactly the right software, and people who want to build on your work can find the latest in your repo. Either know how to cite your work correctly.

In practice drive-by nitpicking over coding style is not that common, particularly in (some) science fields where the other coders are all other scientists who don’t have strong views on it. Nitpicks can be easily ignored anyway.

BTW should you choose to publish, the Turing Way has a section on software licenses written for researchers: https://the-turing-way.netlify.app/reproducible-research/lic...

Re: Ask HN: Should I publish my research code?

#183

In my view and personal experience, the pros outweigh the cons: * You increase the impact of your work and as a consequence also might get more citations. * It's the right thing to do for open and reproducible research. * You can get feedback and improve the method. * You are still the expert on your own code. That someone picks it up, implements an idea that you also had and publishes before you is unlikely. * I nev…

> It's the right thing to do for open and reproducible research. Everybody here talks about how publishing code helps (or even makes possible) reproducibility, but this is not true, on the contrary, it hinders it. Reproducing the results does not mean running the same code as the author and plotting the same figures as in the paper. This is trivial and good for nothing. Reproduction is other researchers independently…

First, this is overtly not true. Reproducibility refers to all forms: that the paper figures can be built from code and don't have errors, that a reimplementation of new code on the same data produces the same results, and that gathering new data (e.g. by conducting the same experiment again if possible, in other words replication) produces comparable results.

Second, publishing code helps make invisible decisions visible in a far better manner than the paper text does. Try as we might to imagine that every single researcher degree of freedom is baked into the text, it isn't and it never has been.

Third, errors do occur. They occur when running author code (stochasticity in models being inadequately pinned down, software version pinning, operating system -- I had a replication error stemming from round-towards-even behaviour implementation varying across platforms). If you have access to the code, then it's far easier to determine the source of the error. If the authors made a mistake cleaning data, having their code makes it easier to reproduce their results using their exact decisions net of the mistake you fix.

Most papers don't get replicated or reproduced. Making code available makes it more likely that, at a minimum, grad students will mechanically reproduce the result and try to play around with design parameters. That's a win.

Source: Extensive personal work in open and transparent science, including in replication; have written software to make researcher choices more transparent and help enable easier replication; published a meta-analysis of several dozen papers that used both reproducing author results from author code, producing author results with code reimplementation, and producing variant results -- each step was needed to ensure we were doing things right; a close friend of mine branched off into doing replications professionally for World Bank and associated entities and so we swap war stories; always make replication code available for my own work.

Re: Ask HN: Should I publish my research code?

#184
While I publish in a field where making source code available is much more common, let me just make a couple of points:

* I have never had someone come back to criticize my code style. And if they do, so what? I'll block them and not think about it again. I don't need to get my feathers ruffled over this.

* Similarly, if someone's trying to replicate my results, and they fail, it's on them to contact me for help. After that it's on me to choose how much effort to put into helping them. But if they don't contact me, or if they don't put in a good faith effort to replicate the results, that's their problem. If they try to publish a failure to replicate without having done that, it's no more valid science than publishing bad science in the first place.

Overall, I think most people who stress about publishing code do so because they haven't done it before. I've personally only ever had good consequences from having done so (people picking up the code who would never have done anything with it if it weren't already open source).

Re: Ask HN: Should I publish my research code?

#185

Earlier quoted context omitted.

I agree, and of course it is. But the underlying issue here is that paper production is the wrong way to do science, there simply isn't enough attention to validate all those papers

That's a good point. However, it suggests to me that one should publish the code for an additional reason: 1) if we acknowledge that paper production is the wrong way to do science, perhaps papers with attached published code are a step in the right direction. A critical mass of papers that are easy to reproduce because anyone can execute the code attached with them could, hypothetically, come to dominate the zeitgei…

Here is an incentive they could use: If a paper publishes the code it used, they can be allowed to skip the methods section about it. Describing code with words is at best awkward and usually error prone as people forget to update the text.

To be clear in my field the journals require publishing the code, but in my experience the code that gets reused gets more scrutiny, often without benefit to the researcher.

Re: Ask HN: Should I publish my research code?

#187

Publishing code could be nice, if for example your code has a commercial application and a company wants to use it, a reference implementation might be nice. Reproducibility -- I dunno. A re-implementation seems better for reproducibility. The paper is the specific set of claim, not the code. If there are built-in assumptions in your code (or even subtle bugs that somehow make it 'work' better), then someone who "rep…

> Really good pseudocode might be more useful than your actual code

Hear hear! OP, if you go this route, treat your implementation as a practice run, and write out exactly how it works in pseudocode.

My 2 cents:

I think that hiring a (good) professional for a rework/reimplementation would be productive, but it would certainly run the risk of exposing errors in your work. If that's desirable or not depends on your timeline to publish, I guess.

Re: Ask HN: Should I publish my research code?

#188
Unambiguously, yes. If possible, release it using some sort of open source license, and grab a DOI for the initial and any subsequent release of the code - you can use Zeonodo or some other tool for this.

I left the academic world a few years ago, but several of the analysis codes/models I published (either as stand-alone tools or artifacts published alongside a journal article) still regularly get used... if anything, there's probably a larger user base for one of my models today than there ever has been, and it's leading to a long-tail of publications where my initial work is either cited or I'm offered co-authorship when I have time to offer hands-on support for improving the model/code and offering my insight as a domain expert.

If you can take the time to clean up some code or author a lightweight package, that's amazing! But it's a bang-for-your-buck type thing. If you ever aspire to leave academia, it's undoubtedly worth spending some time to clean up the code, add documentation, add some unit tests, etc - great artifacts to use in supporting a hiring process if you move into a technical role somewhere in industry. But is far from necessary.

Re: Ask HN: Should I publish my research code?

#189
Consider that "cleaning and organizing" your code means that it is no longer the code that actually produced the results in your paper!

The fact that your code is a mess means that it might be buggy; if other people can see your code, someone might find a bug in it. As you said, this is a good thing for open science, and makes your work easier to reproduce.

Re: Ask HN: Should I publish my research code?

#190
I published some of my Academic code like a tool for simulating superconducting circuits [1] or a tool to manage lab instruments for quantum computing (or other) experiments [2]. It's super niche but both tools have found users in other labs that even keep developing them (at least for [2]). And it's nice to look at your code after 10 years and realize how much you've grown as a programmer :)

[1]: https://github.com/adewes/superconductor [2]: https://github.com/adewes/pyview https://github.com/adewes/python-qubit-setup

Post reply on HN