Be prepared for a metric crapton of crushing silence when you release your code. But do release your code.
Ask HN: Should I publish my research code?
201–210 of 363 posts
Re: Ask HN: Should I publish my research code?
#202> it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc). Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0], making explicit what everyone understands, viz.: "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it int…
This is not what licenses are for!! They are not statements about the quality of your work or anything similar. Use standard and well understood licenses e.g. GPL for code and CC for documentation. The world does not need more license fragmentation.
I agree. There's a lot of confusion surrounding even the most established ones, so there's no need to further muddy the situation with newer licenses. In my opinion a "fun" license, with its untested legal ambiguity, restricts usage more than a well established license with a similar level of freedoms.
Re: Ask HN: Should I publish my research code?
#203Many conferences are starting to adopt a badge system and will evaluate your artifact. And this is becoming more and more popular, and I know many researchers that will keep these badges in mind when reading the evaluation in the paper. For example here is the artifact evaluation that was done at SOSP 2021 https://sysartifacts.github.io/sosp2021/results.html.
Re: Ask HN: Should I publish my research code?
#204Unless you are publishing a software methods paper, you don’t have to worry about cleaning the code or making it portable. In my field, publishing code (and data) is a requirement and has been for years. That doesn’t mean that the code needs to be pretty ( it usually isn’t), it just needs to support the paper.
So, yes. Please publish the code, it will make the rest of the paper stronger.
Re: Ask HN: Should I publish my research code?
#205Let me say that if you do decide to release it it's not just scientists and academics who can stand to benefit. Chances are your paper is less approachable to those outside academia and your code would be easier to understand for an engineer. I would honestly encourage all researchers to publish their code on that basis. You don't have to clean it up or write any scripts to help build it. Just attach what you have and I second the idea to use the CRAPL license!
Re: Ask HN: Should I publish my research code?
#206OP, you shouldn't worry about the state of your code. The could be criticism, but I don't think there's anything that's public and not criticized. A horrible thing that's open source is much better than something that's not. The only real thing to consider here is the type of the license, and weighing the competitive advantage you're talking about. With the license, sites like this[0] can help.
Re: Ask HN: Should I publish my research code?
#207> The paper itself is enough to reproduce all the results.
No, it isn't.
Reproducing the results means that you provide the code that you used so that people can reproduce it just by running "make" (or something similar). If you do not publish the code and the input data, your research is not reproducible and it should not be accepted in a modern, decent world.
It doesn't matter that your code is ugly. Nobody is going to look at it anyway. They are only going to call it. If the code is able to produce the results of the paper with the same input data, that's enough. If the code is not able to at least do that, this means that even you are not able to reproduce your own results. In that case, you shouldn't publish the paper yet.
Re: Ask HN: Should I publish my research code?
#208Yes you should! And not only for ethical reasons (actually reproducible research, publicly financed work, etc), even if those are good enough by themselves.
I've always published my research code. Thanks to that, one of the tools I wrote during my PhD has been re-used by other researchers and we ended up writing a paper together! In my field is was quite a nice achievement to have a published paper without my advisor as a co-author even before my PhD defense (and it most likely counted a lot for me to get a tenured position shortly after).
The tool in question was finja, an automatic verifier/prover in OCaml for counter-measures against fault-injection attacks on asymmetric cryptosystems: https://pablo.rauzy.name/sensi/finja.html
My two most recently published papers also come with published code released as Python package:
- SeseLab, which is a software platform for teaching physical attacks (the paper and the accompanying lab sheets are in French, sorry): https://pypi.org/project/seselab/
- THC (trustable homomorphic computation), which is a generic implementation of the modular extension scheme, a simple arithmetical idea allowing to verify the integrity of a delegated computation, including over homomorphically encrypted data: https://pypi.org/project/thc/
Re: Ask HN: Should I publish my research code?
#209As an amateur who reads journal papers (maybe not the audience you're most concerned about), the two most important things to helping my understanding of the paper's results are, in order:
1. a program that I can run against the data in the paper (where I can modify the data to see how that changes the results the program generates); and
2. the source code to that program, that I can read to understand what it does.
For #1, I'd encourage you to publish something like a Docker image of your built binary, to a permanent public Docker image host; to use that Docker image version of your program to do the actual experiment/data processing for your paper; and then to cite, in your paper, the specific fully-qualified Docker image ID (e.g. hub.docker.com/foo/bar@sha256:abcdef0123...6789) that was used to create the results.
I would also encourage you to, if possible, publish your data in some repository, e.g. GitHub; and to cite the data using a fixed hash (e.g. Git commit hash) as well.
With these two pieces of information, anyone can easily do the simplest possible kind of "reproduction" of your results: namely, they can fetch the same Docker image used in the paper, and then run it against the same data used in the paper, to — hopefully — produce the same results shown in the paper.
---
As for #2...
If you're really worried about "trade secrets", you can just solve #2 by making the code itself only "available upon request."
But don't underestimate the number of people in your field who say they're hoarding their code for reasons of "competitive advantage", but who are really doing so out of personal shame at the state the code is in, and fear that a bug might be found there that will invalidate their result.
These people are, IMHO, not embracing the spirit that led them to become scientists. You should want any bugs in your papers — including in the code — to be found! That's what the pursuit of (academic) science is about — everyone checking each-other's work so that we can all believe more strongly in the results!
You don't need to clean up your code. Maybe get an "alpha reader" to go over it first, like self-published authors do, if you're worried about nitpickers. But the only thing code really "needs" to be valuable, is to compile and run and do something useful.
Personally, all I'd want from your repo is for there to be a Dockerfile in there that will, within its fiddly little internal build environment, manage to output the exact Docker image cited in the paper.
If I cared about modifying the code, I could take the rest from there.
Re: Ask HN: Should I publish my research code?
#210> it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc). Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0], making explicit what everyone understands, viz.: "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it int…
> Academic code is about "proof of concept."
Why does he think that but presumably not the same about the paper itself and the “equations”, plots, etc. contained within?
It’s really not that hard to write pretty good code for prototypes. In fact, I can only assume that he and other professors never allowed or encouraged “proof of concept” code to be submitted as course homework or projects.