Ask HN: Should I publish my research code? • Hacker News Board

Re: Ask HN: Should I publish my research code?

#241

Post by yob22 » Fri, Jan 14, 2022, 6:22 PM UTC

vngzs wrote: post #200

del82 wrote: post #25

> it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc). Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0], making explicit what everyone understands, viz.: "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it int…

Please don't use this license. Copy the language from the preamble and put it in your README if you'd like, but the permissions granted are so severely restricted as to make this license essentially useless for anything besides "validation of scientific claims." It's not an open-source license - if someone wished to include code from CRAPL in their MIT-licensed program, the license does not grant them the permission…

No post body was provided.

Re: Ask HN: Should I publish my research code?

#242

Post by hannob » Fri, Jan 14, 2022, 6:23 PM UTC

bigodanktime wrote: post #203

Many conferences are starting to adopt a badge system and will evaluate your artifact. And this is becoming more and more popular, and I know many researchers that will keep these badges in mind when reading the evaluation in the paper. For example here is the artifact evaluation that was done at SOSP 2021 https://sysartifacts.github.io/sosp2021/results.html .

These badges are kinda controversial. The message they send out is "we give you an extra goodie if you do proper science, because we don't expect that to be the default".

Thus badges can become a kinda excuse for not fixing stuff by default.

Re: Ask HN: Should I publish my research code?

#243

Post by bjourne » Fri, Jan 14, 2022, 6:23 PM UTC

> it will increase the surface for nitpicking and criticism

This is unfortunately. In one of my articles I linked to my github repo where I had implemented the algorithm in C. One of my reviewers complained that I had used C instead of C++. Probably advisable to not publish code before peer review.

Re: Ask HN: Should I publish my research code?

#244

Post by TremendousJudge » Fri, Jan 14, 2022, 6:25 PM UTC

actually_a_dog wrote: post #238

paulluuk wrote: post #137

Earlier quoted context omitted.

They don't call it "Computer Science" for nothing ;)

Maybe a little OT, but, I'd rather it be called "comput ing science." Computers are just the tool. I believe it was Dijkstra who famously objected to it being called "computer science," because they don't call astronomy "telescope science," or something to that effect.

I never thought about this, but it's right. However, to get more nitpicky, most of the uses of "Comput[er|ing] Science" should be replaced with "Computer Engineering" anyway. If you are building something for a company, you are doing engineering, not science research

Re: Ask HN: Should I publish my research code?

#245

Post by bipson » Fri, Jan 14, 2022, 6:27 PM UTC

This whole mindset is so shockingly wrong from an academic perspective.

Research based on or involving code/models/algorithms should always be accompanied by a code drop. Nobody expects the code to be of good quality.

Everything else is not reproducible - and against the scientific codex (IMO).

I read so many papers that claim incredible results, and wondering how they implemented their models in this particular simulator (close to impossible with only what is out there), only to find that there is just nothing to be found, anywhere. No repo, no models, no patch. NIL.

Sending an E-Mail? No response.

Further, anyone could just claim anything this way. Why bother doing any real work?

What if there is a small error in the code?

Wouldn't it be better to know that? In a scientific sense, searching for "the truth"?

Re: Ask HN: Should I publish my research code?

#246

Post by 908B64B197 » Fri, Jan 14, 2022, 6:31 PM UTC

> But on the other hand it's substantially more work to clean and organize the code for publishing

Make sure it's all safe to publish but don't spend any effort on organizing it, unless you can find some grant money for an undergrad to work on it.

If it has users they will contribute their changes to better organize it and use it.

Re: Ask HN: Should I publish my research code?

#247

Post by cycomanic » Fri, Jan 14, 2022, 6:32 PM UTC

bmitc wrote: post #210

del82 wrote: post #25

> it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc). Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0], making explicit what everyone understands, viz.: "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it int…

> Academic code is about "proof of concept." Why does he think that but presumably not the same about the paper itself and the “equations”, plots, etc. contained within? It’s really not that hard to write pretty good code for prototypes. In fact, I can only assume that he and other professors never allowed or encouraged “proof of concept” code to be submitted as course homework or projects.

I think you don't understand the realities of work in scientific/academic organisations. Unless you work in computer science you likely never received any formal education on programming except for some numerical methods in c/matlab/fortran course during your studies (which often also focus more on the numerical methods and not the programming). So pretty much every person, just learned by doing.

Moreover you are not being paid for writing reasonable programs you're paid for doing science. Nobody would submit "prototype" papers, because they are the currency of academic work. There is lots of time spend on polishing a paper before submission, but doing that for code is generally not appreciated because nobody will see this on your CV.

Re: Ask HN: Should I publish my research code?

#248

Post by dm319 » Fri, Jan 14, 2022, 6:36 PM UTC

grahamlee wrote: post #182

Hi, I’m a Research Software Engineer (a person who makes software that helps academics/researchers) at a university in the UK. My recommendation is that not only do you publish the code, you mint a DOI (digital object identifier, Zenodo is usually the go to place for that) for the specific version that was used in your paper and you associate them. And you include a citation file (GitHub supports them now: https://do…

For a physician that writes pretty awful code, I like this comment.

Re: Ask HN: Should I publish my research code?

#249

Post by deadmutex » Fri, Jan 14, 2022, 6:37 PM UTC

guy-brush wrote: post #56

In my view and personal experience, the pros outweigh the cons: * You increase the impact of your work and as a consequence also might get more citations. * It's the right thing to do for open and reproducible research. * You can get feedback and improve the method. * You are still the expert on your own code. That someone picks it up, implements an idea that you also had and publishes before you is unlikely. * I nev…

> It's great for your CV. Companies love applicants with open-source code.

While I strongly support sharing the code, I am not sure if this is a great reason to do so. Companies are made up of many individuals, and while some might appreciate what it takes to open source code, other individuals might judge the code without full context and think it is sloppy. My suggestion is that you fully explain the context before sharing code with companies.

Re: Ask HN: Should I publish my research code?

#250

Post by webmaven » Fri, Jan 14, 2022, 6:38 PM UTC

albertzeyer wrote: post #197

> The paper itself is enough to reproduce all the results. No, this is almost never the case. It should be. But it cannot really be. There are always more details in the code than in the paper. Note that even the code itself might not be enough to reproduce the results. Many other things can matter, like the environment, software or library versions, the hardware, etc. Ideally you should also publish log files with a…

> And random seeds. Make sure this part is at least deterministic by specifying the seed explicitly (and make sure you have that in your log as well).

> Unfortunately, in some cases (e.g. deep learning) your algorithm might not be deterministic anyway, so even in your own environment, you cannot exactly reproduce some result. So make sure it is reliable (e.g. w.r.t. different random seeds).

Publishing the weights of a trained model allows verification (and reuse) of results even before going to the effort of reproducing it. This is especially useful when training the model is prohibitively expensive.