Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Should I publish my research code?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 363 posts

Re: Ask HN: Should I publish my research code?

#131

Post by UncleOxidant »

Emphatically YES. Put the code on GitHub. It doesn't have to be perfect. Especially if it will take two months for someone to "get it right" from the paper. I've been involved in projects where we were trying to reproduce results from some paper both with code and without. The description of an algorithm in a paper can sometimes be unclear, often reading the code makes the description in the paper much clearer. In the cases where no node code is provided it's that much harder to reproduce the results. You want to make it as easy as possible for others to reproduce your results - give them your code - put it in a github repo. If they spot discrepancies between the code and what's described in the paper, then all the better - you can use that feedback to improve both.

I'll add: I think that we need to change the mindset in academia about code. If code was involved in producing the results in the paper that code should be considered part of the paper and (at least) as important as the text of the paper. (Same for data)

Re: Ask HN: Should I publish my research code?

#132

Post by goodpoint »

nomilk wrote: post #17

> it will increase the surface for nitpicking and criticism Anyone who programs publicly (via streaming, blogging, open source) opens themselves up for criticism, and 90% of the time the criticism is extremely helpful (and the more brutally honest, the better). I recall an Economist magazine author made their code public, and the top comments on here were about how awful the formatting was. The criticism wasn't unwar…

> 90% of the time the criticism is extremely helpful

Citation needed. I have rarely seen valuable feedback from random visitors from the internet.

Re: Ask HN: Should I publish my research code?

#134

Post by thayne »

My 2 cents: you should publish the code as you used it in your research, so that it's possible to review your code. If there is a bug in your code, that could impact your results, and that problem would be much harder to find/reproduce without your source code.

Re: Ask HN: Should I publish my research code?

#135

Post by cjekel »

So it really depends upon what you want out of your research career. Part of being a successful researcher is making an impact on the community. This involves producing works that the community finds useful. I've always looked at making my code available as another avenue to help increase the impact of my work. In my case, many more people have used my public codes than have ever read my papers.

You have limited time. I'd prioritize that time on what you think others will find useful.

Don't worry about ugly code. There are research codes with 1k+ stars on GitHub that are ugly. They have so many stars because people find them useful.

You absolutely don't have to publish your code, or anything else of that matter. Don't let the the drive for impact on the community force you into working on something you're not interested in.

Congrats on your publication.

Re: Ask HN: Should I publish my research code?

#137

Post by paulluuk »

LandR wrote: post #98
del82 wrote: post #25

> it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc). Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0], making explicit what everyone understands, viz.: "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it int…

> "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it into running; and it's not pretty code. Academic code is about 'proof of concept.'" What do you know, it turns out the professional software developers I work with are actually scientists and academics!!

They don't call it "Computer Science" for nothing ;)

Re: Ask HN: Should I publish my research code?

#138

Post by bee_rider »

del82 wrote: post #25

> it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc). Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0], making explicit what everyone understands, viz.: "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it int…

I like it so far, other than

4) You recognize that any request for support for the Program will be discarded with extreme prejudice.

I think that should be a "may" rather than a "will." If I find out someone is using my obscure academic code, and they ask for help, I'd be pretty pumped to help them (on easy requests at least).

Re: Ask HN: Should I publish my research code?

#139

Post by dabreegster »

anon_123g987 wrote: post #108
guy-brush wrote: post #56

In my view and personal experience, the pros outweigh the cons: * You increase the impact of your work and as a consequence also might get more citations. * It's the right thing to do for open and reproducible research. * You can get feedback and improve the method. * You are still the expert on your own code. That someone picks it up, implements an idea that you also had and publishes before you is unlikely. * I nev…

> It's the right thing to do for open and reproducible research. Everybody here talks about how publishing code helps (or even makes possible) reproducibility, but this is not true, on the contrary, it hinders it. Reproducing the results does not mean running the same code as the author and plotting the same figures as in the paper. This is trivial and good for nothing. Reproduction is other researchers independently…

In my experience, the published paper is super vague on the approach, and implementing it without further references is really hard. I'm not necessarily arguing that papers should get longer and more detailed to counter this; expressing the details that matter in code seems like a more natural way to communicate anyway.

Why trust results if you can't see the methodology in detail and apply the approach to your own data? I once knew somebody who built a fuzz tester for a compilers project, got ahold of a previous project's compiler code that won best paper awards, and discovered a bunch of implementation bugs that completely invalidated the results.

Why is the peer review process limited to a handful of people who probably don't have access to the code and data? If your work is on Github, anybody can come along and peer review it, probably in much more detail. And as a researcher, you don't get just one chance to respond to their feedback -- you can actually start a dialogue, which other people are free to join in.

As long as a project's README makes any sort of quality / maintenance expectations clear upfront, why not publish your code?

Re: Ask HN: Should I publish my research code?

#140

Post by jll29 »

Publishing the code is great, as some of the questions a reader of your paper may have can only answered by looking into the source code, as no paper has enough space to talk about all implementation details in a real-life complex system.

There is value in scrutinizing the code - not w.r.t. coding styles or standards but to discover bugs in the implementation, which are very common. Scientists are only human, and scientific software is less often checked by a second pair of eyes. There is also value in trying to replicate a study from scratch with a fresh implementation only from the details in the paper. Many conferences, for instance the European Conference on Information Retrieval (ECIR), Europe's largest scientific search technology conference, has a replication track only for replication papers, and these are often the most interesting/insightful papers. It occasionally happens that a result is not caused by what the authors think, but is merely an artifact of the implementation code. A very famous MIT researcher (not naming him or her here on purpose) fell into this trap in their Ph.D. thesis, but it can happen to anyone, really. Scientific results become objective knowledge as others solidify the body of knowledge by carrying out replications and arriving at the same results.

Whatever your decision about past code, going forward, if you plan to release all future research code, you will likely write better code in the first place, as you will constantly be aware that people will be looking at it, and that can only be a good thing.

Post reply on HN