Live data from Hacker News

Ask HN: Should I publish my research code?

news.ycombinator.com

301–310 of 363 posts

Re: Ask HN: Should I publish my research code?

#301

> The paper itself is enough to reproduce all the results. No, this is almost never the case. It should be. But it cannot really be. There are always more details in the code than in the paper. Note that even the code itself might not be enough to reproduce the results. Many other things can matter, like the environment, software or library versions, the hardware, etc. Ideally you should also publish log files with a…

I agree with almost all of this, however I believe that publishing random seeds is dangerous in its own way. Ideally, if your code has a random component (MCMC, bootstrapping, etc), your results should hold up across many random seeds and runs. I don’t care about reproducing the exact same figure you had, I want to reproduce your conclusions. In a sense, when a laboratory experiment gets reproduced, you start off wit…

The thing is, if you want to ignore someone's random seed, you can if it's provided. If it's not provided and you need it to chase down why something isn't working, you're SOL.

It's zero cost to include it.

Re: Ask HN: Should I publish my research code?

#302
Here are my checklist to publishing research codes:

1. encumbered by pending or active patent(s)?

2. release of proprietary holds by corporates or participants

3. any tangible market values worth pursuing, then keep it to yourself.

4. any conflict with trademarks, copyrights, or domain hold? Rename it

that’s just some of the points. Contact your local VCs if it has any traction.

Re: Ask HN: Should I publish my research code?

#303
In my field (bio) it's fairly common not to publish code, but it's becoming more common. Biologist's code is generally crappy and I think everyone understands that. The better developers are often valued for producing tools that are reliable and people can use and get lots of attention and citations for their tool papers.

The mathematicians and computer scientists I've worked with generally wrote more complicated code, but from a bugginess and maintainability standpoint I'm not sure it was any better. I had a mentor with an applied math degree who was extremely fond of one and two character variable names.

Just publish it. Unless your paper is a _BIG_DEAL_ barely anybody is going to look at it, and some people (hopefully the right people) will respect you for showing your work. I think I'm one of the few reviewers that actually try to run and maybe glance at the code for papers I review. In the papers I've reviewed I've never seen a comment that indicated any of the other reviewers even looked at it.

Re: Ask HN: Should I publish my research code?

#304

Earlier quoted context omitted.

notoriously Philip Wadler says that computer science has two problems: computer and science. It's not about computers and "You don't put science on your name if you're a real science" He prefers the name informatics. source: https://youtube.com/watch?v=IOiZatlZtGU

"Information science" is basically long form of "informatics" so that breaks it I'd say. Also, "information" tends to imply a focus on state and places computational aspects (operations performed on information) as second hand. I've yet to find a classification I really like but this is an interesting take. I still tend to like CIS (Computing and Information Sciences). The problem with CS is it focuses on computation…

TBF informatics is "the study of computational systems"

I have studied informatics, but don't call myself an informatic, because I am not doing any research.

I call myself a programmer, because I am not doing engineering either.

Re: Ask HN: Should I publish my research code?

#305

Earlier quoted context omitted.

This has explicit usage limitations that matter in science land, which is very much the kind of thing that belongs in a license. Eg: You are permitted to use the Program to validate scientific claims submitted for peer review, under the condition that You keep modifications to the Program confidential until those claims have been published. Moreover, sure, lots of the license is text that isn't common in legal docume…

> clearly communicate intent Again, this is not how a license work. You can express your intents, ideas and desires in a README file and in many other ways. The license is nothing more than a contract that provides rights to the recipient under certain conditions. Standing up in court is its real power and only purpose. That's why we should prefer licenses that stood up in court and have been written by lawyers rathe…

I strongly disagree. Contracts very much primarily communicate intent, ideally in such a way that they also stand up in court. People regularly argue over details in contracts, people regularly look up things in contracts, also when there is no court to be seen and no intention anywhere to go to court. The vast vast vast majority of contracts never make it to court.

Plenty of contracts aren't even written down. When you buy a loaf of bread at the bakery, you make an oral contract about that transaction.

The idea that contracts, or licenses, need to be written in dull legalese and be pretty much impenetrable to be useful or "valid" or whatever, is absolutely bonkers. Lawyers like you to think that but it's not true. It's an urban legend.

If you need to make sure that you can defend your rights in court, then sure, you're probably going to need some legalese (but even then there's little harm in also including some intent - it's just not very common). Clearly that's not the goal here. No scientist is gonna sue another scientist who asked for support and got angry about not getting any even though the code was CRAPL licensed.

Re: Ask HN: Should I publish my research code?

#306
You could also paint a picture no one else will ever see.

Personally, I hate it when academics do not publish their code. Some academics publish the code but not the pretrained model or withhold the dataset, to collect dust on their computer.

People who publish code, datasets and models become the core building blocks of future work. People who don't fade away people do not remember their names.

Re: Ask HN: Should I publish my research code?

#307
If you think you've got something that will give you a competitive advantage, seize that advantage. Otherwise, it will be no more than source code on a thumb drive that you'll eventually forget the encryption password to, or gets damaged during a move, or is lost when you stop paying for your cloud storage membership, is lost when you re-partition the wrong part of your hard drive, whatever.

Re: Ask HN: Should I publish my research code?

#308
Absolutely, yes. The other comments here have some fantastic reasons for doing this, and several do a good job of weighing the pros vs cons.

The paper alone is, almost always, never enough to fully reproduce the result. I've been bitten by this almost every time I've tried to implement someone else's computational model. It comes down to that only relying on your paper to explain your code leaves a LOT of room for errors. I've experienced all of these when trying to implement someone else's computational work without their code being published:

    1. Despite your best efforts, you include fundamental, result-breaking typos in the equations you write up to explain the math of what you're doing. This WILL happen to you at some point in your career, and in my experience, it's a problem in >>50% of computational modeling papers.
    2. There are assumptions in the logic of the code that you don't include in the writeup, since they're obvious to you, but you don't realize that someone else trying to understand your paper won't necessarily be starting with those same assumptions. This happens frequently with neural models that use complicated synapse-computation schemes.
    3. Your codebase may be big enough that you think code part X works a certain kind of way from memory, but you forget that you changed the logic late in the project to work in a different way.
    4. Publishing your code at the time of publication prevents "Which version did I use?" problems. It's very common for people to continue to work on their science code for new work, but they don't bother to save/tag a SPECIFIC version of their code that was used for the actual paper. This results in that even the author doesn't know what exact values were used for the results in the paper!
Any "competitive advantage" has to be weighed versus "positive exposure". If your code is the primary research object (as opposed to the data), then it's technically possible that someone may grab your code, extend it to do the next, interesting use of it, and then scoop you before you can do it yourself. However, even if this happens (which it probably won't), consider the following:

    1. You can't build a successful career out of just small extensions to the same piece of code, and so that codebase won't be the main kernel of your career, but rather your understanding of it.
    2. For every 1 person that tries to use that to scoop you, IMHO there's going to be at least 10 other people who see your code and reach out to you for help with it, or just to ask a question about it, or reach out for potential collaboration! In other words, depending on the field, if you publish the code, I think you're likely to gain new/future collaborators at a MUCH faster rate than people who compete against you. You'll be surprised at how many researchers on the other side of the planet are interested in your software!
    3. Even if someone scoops you with your own code, if they give any indication it came from you, you still get to count that as a publication that built off of your software work when you're applying to jobs :)
    4. At least with US federal government funding, it's gradually becoming required to do this anyways, and I believe/hope that it's going to become the standard anyways very soon.
Finally, don't fret about polishing/cleaning/organizing the code, especially style. For others trying to reproduce your results or just investigating how you did things, the main thing that matters is that your code runs "correctly", i.e. how you ran it to get the results that you did. One idea is to publish it "as is" for the CORRECTNESS of the paper, put a git tag indicating "original version", and THEN clean it up on Github/wherever. This helps prevent any new "organizing" of the code from potentially breaking something, which is counterproductive. This way, when people go to your code page, the first thing they see is a nicely-organized version, and gives you time to test that it works the same. Honestly, if you care enough about this at all, then your code is probably significantly more organized than 95% of research code out there; the standards of code quality in science are VERY low, which is completely different than private sector software engineering.

* edits are for markup

Re: Ask HN: Should I publish my research code?

#309
Yes, absolutely.

You should have confidence in the correctness of your code if you are publishing.

If your code is a shitshow, why do you trust it? Decent code is to your own advantage even if no one else ever looks at it.

In the best case, it’s possible to build a community around your code, to wide benefit and your career benefit. I’ve seen this with several peers and students.

As a hiring manager, it’s very nice indeed to read a paper and scan the code of an fresh grad applicant.

My lab’s approach is to put the repo in public and put the hash of the relevant commit in the paper. Then you can keep developing there but readers can be confident they can get the exact code used to justify the claims in the paper.

An exception is if you plan to make a company around your IP. You should estimate how likely this is to happen before defaulting to this.

Post reply on HN