It's sort of funny when the pro list includes "better for science" and there's still a need for a con list. There should be a scientific equivalent to the hypocratic oath; a lot of us laypeople imagine that scientist default to "good for science" and "ease and possibility of replicatability."
Ask HN: Should I publish my research code?
91–100 of 363 posts
Re: Ask HN: Should I publish my research code?
#92I am not a researcher in the sense that I'm not publishing papers but I'm a consumer of research. Every day I can find the source code for the paper is a great day. Even if it's some language I don't use I still have something to go off of. Often it's easier to red some code to understand the method than to read the paper itself. I'm used to read code. I do it almost everyday and I'm relatively proficient at it. I'm not very well at untangling academic language or having to read 30 years worth of papers to get all the assumptions made in a paper.
As an example, I've found a paper that promises a method to do the very thing I want to accomplish. It's not too dense but it skips a few crucial moments and I've been working on coding the method for a year now (on and off, of course but still for a long time). If the code was available it probably wouldn't take as long. The paper didn't mention that the code was available upon request but it was implemented in a piece of software. I've found it eventually but it was a version just before the feature I'm after was added. I tracked the author and they were great sport about cold emails bet didn't have the source any more.
So yes, please publish the code. You don't have to clean it up. It worked for the paper — it's good enough. Even the most terrible code is immeasurably better than no code.
Re: Ask HN: Should I publish my research code?
#93If you publish it, please make sure it works today, and can work tomorrow. Pin the versions of any dependencies, or bundle them if feasible. Also, include basic instructions for running your code. I helped my wife with a replication study that should have been straightforward, and I was unable to get the code running after about a week. I don’t necessarily believe the research was suspect, but broken code does draw m…
Nice to have. But it is often the case that it is often difficult to avoid things like hard-coded paths, system-specific environment variables, etc. that simply won't translate to another environment. Most of the times, those things don't interfere with comprehension.
Re: Ask HN: Should I publish my research code?
#94Most code in the world is crap code, so don't worry, putting it out there lets people make it better
Re: Ask HN: Should I publish my research code?
#95Depends on the field. Let's assume you are not in math /CS/Physics
What can go "wrong"
- Someone may find a minor rounding error and now you have to issue a correction to the paper which, laudable as it is, is a bad thing
- You 'll end up having to maintain an open-source-something and possibly forks
- Your open source code may end up as a github repo in which you are just one of the contributors, not the owner and others are leeching credit from u
- People who want to criticise you will find excuses in the coding style.
Research code is messy -- it must be messy imho, or else it's probably insignificant. People who don't publish it are definitely shielded by the obscurity , while i have received scrutiny for entirely inconsequential details. You can choose to publish it in a less accessible way , which will thwart people with bad intentions. Even publishing it as a tarball in a web server is enough work to keep them away.
Re: Ask HN: Should I publish my research code?
#96FWIW, this is how I've released the crappy barely-working "academic quality" code for a paper in the past:
https://github.com/DarwinAwardWinner/cd4-histone-paper-code
The main points are that I made only a minimal attempt to organize it, and I made the state of the code clear in the README. I don't recall anyone complaining about the code or even mentioning it during review. (Though to be fair, I also don't recall whether I published the code before or after the paper was accepted.)
Looking at things from the other side, I'm am at least an order of magnitude more likely to read, use the work/methods from, and therefore cite a paper that comes with code.
Re: Ask HN: Should I publish my research code?
#97> The paper itself is enough to reproduce all the results.
I've heard this claim so many times, from many an author who had their brain so deep in the problem they were working on that they were 100% incapable of properly gauging the validity of this claim.
To verify that what you claim is true, wait two years to give your brain time to flush the context, pick your research paper up (and nothing else that wasn't made available to others) and try to reproduce the results on a brand new computer without any of the environment your developed your research with.
See how much blood you end up sweating.
PLEASE publish your research code. Don't worry about it being disgusting and hackish, it's research code, so by definition, no one expects it to be industrial strength.
Don't spend time cleaning it up either, your time is better spent on doing more research.
If you feel responsibility towards the community:
- put a huge disclaimer at the start of the README explaining what a mess the whole thing is *because* it's research code.
- if your really must: list requirements and provide a build.shRe: Ask HN: Should I publish my research code?
#98> it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc). Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0], making explicit what everyone understands, viz.: "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it int…
> "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it into running; and it's not pretty code. Academic code is about 'proof of concept.'"
What do you know, it turns out the professional software developers I work with are actually scientists and academics!!
Re: Ask HN: Should I publish my research code?
#99Earlier quoted context omitted.
Thanks, agreed. Small note: it is not clear what Minion is doing, from just visiting the github repo. Perhaps add "C++ constraint solver" in the github description, but it is still unclear: it could be a rigid body constraint solver for games? Maybe add a link to a paper?
Yes, I should practice making things more accessible :) In practice Minion is generally used as a backend to Conjure ( https://conjure.readthedocs.io/en/latest/ ), which provides a much nicer input language.
Thanks, I was not familiar with Conjure and general Constraint Programming. I haven't seen it in real-time appications for games or robotics (usually highly optimized domain specific constraint solvers are used there, for rigid body, fluid sim, cloth, deformables etc)
Re: Ask HN: Should I publish my research code?
#100Besides all the pros already mentioned, there is a high chance other researchers will use your code and cite you. If they have to compare their results with some previous state of the art, it will be the one with available code. The whole thing about “the paper is enough to reproduce” never happens, ever.