Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Should I publish my research code?

news.ycombinator.com

Post reply on HN
Print view View original

221–230 of 363 posts

Re: Ask HN: Should I publish my research code?

#221

Post by grst »

Releasing the code is the very least you should do to make your analysis reproducible. I would be surprised if it was possible to exactly reproduce the results from the paper alone.

From Heil et al. (https://www.nature.com/articles/s41592-021-01256-7):

> Documenting implementation details without making data, models and code publicly available and usable by other scientists does little to help future scientists attempting the same analyses and less to uncover biases. Authors can only report on biases they already know about, and without the data, models and code, other scientists will be unable to discover issues post hoc.

Even better would be to containerize all software dependencies and orchestrate the analysis with a workflow manager. The authors of the above paper refer to that as "gold standard reproducibility"

Re: Ask HN: Should I publish my research code?

#222

Post by ironrabbit »

As someone who has had to reproduce others' research results: it is much much better to release the unclean, unorganized code that actually produced your results than it is to release nothing. Even if it doesn't run (e.g. it depends on a hardware system that the user won't have access to) it's still better for people to be able to read your code and understand some tricky part that isn't fully explained in your paper.

Re: Ask HN: Should I publish my research code?

#224

Post by analog31 »

enriquto wrote: post #207

> The paper itself is enough to reproduce all the results. No, it isn't. Reproducing the results means that you provide the code that you used so that people can reproduce it just by running "make" (or something similar). If you do not publish the code and the input data, your research is not reproducible and it should not be accepted in a modern, decent world. It doesn't matter that your code is ugly. Nobody is goin…

Worth noting that this is a modernization of scientific tradition, which predates code. I did publish my code, but the bulk of my work was a series of manual steps. That was 30 years ago. The closest thing to a replication involved changes to my design. The science world at large is still coming up to speed on this.

Re: Ask HN: Should I publish my research code?

#225

Post by mattbaker »

In undergrad I was so grateful whenever a CS paper came with code. It helped me learn and comprehend so much better and I always wanted to thank the person who did it (sometimes I even did if their email was there :)).

You might be doing a young student a solid :D And don’t worry about cleaning it up!

If you use GitHub you could even disable Issues and have a note saying you don’t accept pull requests (in case you’re worried about support burden).

Re: Ask HN: Should I publish my research code?

#226

Post by kube-system »

del82 wrote: post #25

> it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc). Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0], making explicit what everyone understands, viz.: "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it int…

MIT and BSD are established and well accepted licenses, literally named after the academic institutions where they originated. Licenses are legal documents, part of what makes them "explicit what everyone understands" is their legally recognized establishment.

If you want to set expectations, this can simply be done in a README. Putting this in a license makes no sense. Copyright licenses grant exceptions to copyright law. If you're adding something else to it, you're muddying the water, not making it better.

Re: Ask HN: Should I publish my research code?

#227

Post by Rayhem »

tsumnia wrote: post #211
anon_123g987 wrote: post #108

Earlier quoted context omitted.

> It's the right thing to do for open and reproducible research. Everybody here talks about how publishing code helps (or even makes possible) reproducibility, but this is not true, on the contrary, it hinders it. Reproducing the results does not mean running the same code as the author and plotting the same figures as in the paper. This is trivial and good for nothing. Reproduction is other researchers independently…

> Reproducing the results does not mean running the same code as the author and plotting the same figures as in the paper. This is trivial and good for nothing. I agree with this statement, however I think you may have a misunderstanding on reproducing results. It's not that you can reproduce their graphs from their dataset, but rather seeing if their code reproduces on to your (new) dataset. Another way to think of…

I guess things are a spectrum. I've worked on research projects where understanding and developing the algorithm is the research. There isn't really an "input data set" other than a handful of parameters that are really nothing more than scale factors, and the output is how well the algorithm performs. So "setting up the laboratory" by cloning the code and building it is...fine, but a reimplementation of the algorithm with "the same" results (modulo floating point, language features, etc. etc.) aligns much better with reproducibility.

Re: Ask HN: Should I publish my research code?

#228

Post by goodpoint »

jll29 wrote: post #129
goodpoint wrote: post #101

Earlier quoted context omitted.

This is not what licenses are for!! They are not statements about the quality of your work or anything similar. Use standard and well understood licenses e.g. GPL for code and CC for documentation. The world does not need more license fragmentation.

For instance, the Java license explicitly forbids the use in/for real-time critical systems, and such limitations are good to stress in a license so that they may reach legal force, also to protect the author(s). Incidentally, I've seen people violate the Java "no realtime" clause.

And it makes the license non-opensource.

Plus, the usual "no warranty" is strong enough to protect the authors anyways.

Re: Ask HN: Should I publish my research code?

#230

Post by danuker »

vngzs wrote: post #200
del82 wrote: post #25

> it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc). Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0], making explicit what everyone understands, viz.: "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it int…

Please don't use this license. Copy the language from the preamble and put it in your README if you'd like, but the permissions granted are so severely restricted as to make this license essentially useless for anything besides "validation of scientific claims." It's not an open-source license - if someone wished to include code from CRAPL in their MIT-licensed program, the license does not grant them the permission…

> the permissions granted are so severely restricted as to make this license essentially useless

Indeed, also there're things like "By reading this sentence, You have agreed to the terms and conditions of this License.". That can't hold up in court! How can I know in advance what the rest of the conditions say before agreeing to them?

Then again, I am not a lawyer either.

Post reply on HN