Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Should I publish my research code?

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 363 posts

Re: Ask HN: Should I publish my research code?

#151

Post by allenofthehills »

anon_123g987 wrote: post #108
guy-brush wrote: post #56

In my view and personal experience, the pros outweigh the cons: * You increase the impact of your work and as a consequence also might get more citations. * It's the right thing to do for open and reproducible research. * You can get feedback and improve the method. * You are still the expert on your own code. That someone picks it up, implements an idea that you also had and publishes before you is unlikely. * I nev…

> It's the right thing to do for open and reproducible research. Everybody here talks about how publishing code helps (or even makes possible) reproducibility, but this is not true, on the contrary, it hinders it. Reproducing the results does not mean running the same code as the author and plotting the same figures as in the paper. This is trivial and good for nothing. Reproduction is other researchers independently…

This is a common misconception, but you are actually talking about "replicability" which is "writing and then running new software based on the description of a computational model or method provided in the original publication, and obtaining results that are similar enough" [1]. Reproducibility instead refers to running the same code on the same data to get the same results [2].

[1] Rougier et al., "Sustainable computational science: the ReScience initiative" https://arxiv.org/abs/1707.04393

[2] Plesser, "Reproducibility vs. Replicability: A Brief History of a Confused Terminology" https://doi.org/10.3389/fninf.2017.00076

Re: Ask HN: Should I publish my research code?

#152

Post by orforforof »

I would agree with many others here who say publish it. In some fields there is an additional question of where to host it, lest your paper's impact outlast the lifetime of your current GitHub repo or whatever. There are good solutions out there. Assuming you are at a university it's worth having a chat with a librarian.

Re: Ask HN: Should I publish my research code?

#153

Post by jerven »

jll29 wrote: post #129
goodpoint wrote: post #101

Earlier quoted context omitted.

This is not what licenses are for!! They are not statements about the quality of your work or anything similar. Use standard and well understood licenses e.g. GPL for code and CC for documentation. The world does not need more license fragmentation.

For instance, the Java license explicitly forbids the use in/for real-time critical systems, and such limitations are good to stress in a license so that they may reach legal force, also to protect the author(s). Incidentally, I've seen people violate the Java "no realtime" clause.

Used to, OpenJDK is licensened under GPLv2 with the classpath excemption that allows this for years. If not running an OpenJDK build it depends on your vendor license.

Re: Ask HN: Should I publish my research code?

#154

Post by cblconfederate »

shadowgovt wrote: post #150
cblconfederate wrote: post #95

Depends on the field. Let's assume you are not in math /CS/Physics What can go "wrong" - Someone may find a minor rounding error and now you have to issue a correction to the paper which, laudable as it is, is a bad thing - You 'll end up having to maintain an open-source-something and possibly forks - Your open source code may end up as a github repo in which you are just one of the contributors, not the owner and o…

Isn't it worse for everyone if the paper but not the code is published and somebody doesn't find the error? The world suffers for acting off of untrue information, and eventually the author suffers for having been wrong all this time.

I agree, and of course it is. But the underlying issue here is that paper production is the wrong way to do science, there simply isn't enough attention to validate all those papers

Re: Ask HN: Should I publish my research code?

#155

Post by anon_123g987 »

dabreegster wrote: post #139
anon_123g987 wrote: post #108

Earlier quoted context omitted.

> It's the right thing to do for open and reproducible research. Everybody here talks about how publishing code helps (or even makes possible) reproducibility, but this is not true, on the contrary, it hinders it. Reproducing the results does not mean running the same code as the author and plotting the same figures as in the paper. This is trivial and good for nothing. Reproduction is other researchers independently…

In my experience, the published paper is super vague on the approach, and implementing it without further references is really hard. I'm not necessarily arguing that papers should get longer and more detailed to counter this; expressing the details that matter in code seems like a more natural way to communicate anyway. Why trust results if you can't see the methodology in detail and apply the approach to your own da…

> In my experience, the published paper is super vague on the approach, and implementing it without further references is really hard.

This is my experience, too, and in my opinion this is exactly what has to change for really reproducible research, not ready to run software supplied by the author.

There are many good arguments in support of publishing code, but reproducibility is not one of them, that's all I'm saying.

Re: Ask HN: Should I publish my research code?

#156

Post by shmageggy »

> Besides, many scientists look at code as a competitive advantage so in this case publishing the code will be removing the competitive advantage.

This is probably wrong, depending on the field. At least in machine learning, the papers that get cited the most are those that other people can easily pick up and work on. They become the basis for future work, get cited as baselines more often, etc. Publishing research ML code is a competitive advantage.

Re: Ask HN: Should I publish my research code?

#157

Post by stickfigure »

rvanlaar wrote: post #40

What a great question. You've come to the right community. My main concern would be to make sure there are no passwords or secret keys in the data, not how it looks. You'll open yourself up for comments. They may be positive or negative. You'll only know how it pans out afterwards. Is the code something that you'll want to improve on for further research? If so publish it on github. It opens the way for others to con…

> My main concern would be to make sure there are no passwords or secret keys in the data, not how it looks.

Worth mentioning specifically: If you make a git (et al) repository public, make sure there are no passwords or secret keys in the history of the repository either. Cleaning a repository history can be tricky, so if this is an issue, best to just publish a snapshot of the latest code (or make 100% sure you've invalidated all the credentials).

Re: Ask HN: Should I publish my research code?

#158

Post by pottertheotter »

Icko wrote: post #11
PragmaticPulp wrote: post #2

Is your goal to help advance the science and our general knowledge? Publish the code. You don’t even need to clean it up. Just publish. Don’t worry about coding style nitpicks. Having the code and data available actually protects you from claims of fabrication or unseen errors in hidden parts of your research. On the other hand, if your goal is only to advance your own career and you want to inhibit others from opera…

Not sure if posting the paper only is even the best move. I personally never work with papers with no code published. Just not worth the effort to reproduce them, when I can use the second SOTA for nearly no performance penalty and much less effort. All the groundbreaking papers in deep learning in the last decade had code published. So if you're aiming for thousands of citations, you need code.

This is very domain specific. OP said it is not the norm to do publish code in his field. I have a PhD and in my field it is the same. So much so that I can't think of any paper in my field that has code published. Therefore, a paper with no code would not be at a disadvantage.

Personally, it is a pet peeve I have about my field. But there is no incentive for a new researcher to publish code as it decreases barriers to entry. As much as it's nice to say that researching in academia is about progressing science, as a researcher, you are your own startup trying to make it (i.e., get tenure).

Re: Ask HN: Should I publish my research code?

#159

Post by UncleOxidant »

anon_123g987 wrote: post #108
guy-brush wrote: post #56

In my view and personal experience, the pros outweigh the cons: * You increase the impact of your work and as a consequence also might get more citations. * It's the right thing to do for open and reproducible research. * You can get feedback and improve the method. * You are still the expert on your own code. That someone picks it up, implements an idea that you also had and publishes before you is unlikely. * I nev…

> It's the right thing to do for open and reproducible research. Everybody here talks about how publishing code helps (or even makes possible) reproducibility, but this is not true, on the contrary, it hinders it. Reproducing the results does not mean running the same code as the author and plotting the same figures as in the paper. This is trivial and good for nothing. Reproduction is other researchers independently…

Often the text in a paper that describes some algorithm will not be completely clear or complete. Providing the code fills in those blanks. I've taken a paper with code and gone through the code line by line comparing it with what was described in the paper. The code often ends up clarifying ambiguities. In some cases there's an outright disagreement between the paper and the code - that's where the feedback comes in, ask the author about these disagreements, it will help them clarify their approach (both the text and the code).

> Reproducing the results does not mean running the same code as the author and plotting the same figures as in the paper.

Sure, to some extent. But the code does provide a baseline, a sanity check. People who are trying to reproduce results should (as I describe above) go through both the paper and the code with a fine tooth comb. The provided code should be considered a place to start. I'll often re-implement the algorithm in a different language using the paper and the provided code as a guide.

Re: Ask HN: Should I publish my research code?

#160

Post by shadowgovt »

cblconfederate wrote: post #154
shadowgovt wrote: post #150

Earlier quoted context omitted.

Isn't it worse for everyone if the paper but not the code is published and somebody doesn't find the error? The world suffers for acting off of untrue information, and eventually the author suffers for having been wrong all this time.

I agree, and of course it is. But the underlying issue here is that paper production is the wrong way to do science, there simply isn't enough attention to validate all those papers

That's a good point. However, it suggests to me that one should publish the code for an additional reason:

1) if we acknowledge that paper production is the wrong way to do science, perhaps papers with attached published code are a step in the right direction. A critical mass of papers that are easy to reproduce because anyone can execute the code attached with them could, hypothetically, come to dominate the zeitgeist and push out papers with grandiose but unverified claims.

Post reply on HN