Let me see your papers, let me see your source
cflewis.posterous.com
Let me see your papers, let me see your source
1–10 of 32 posts
Re: Let me see your papers, let me see your source
#2Reminds me of my own experience doing research in mobile robotics. It basically followed a cycle described pretty well by the the comic here: http://www.willowgarage.com/sites/default/files/blog/201004/...
Re: Let me see your papers, let me see your source
#3Open-sourcing is not enough. Honest researchers should also publish all data sets for peers to validate results. When I was in grad school, it was a disappointing fact that very few academics in the machine learning field did this.
Re: Let me see your papers, let me see your source
#4I think there are other motivations involved here. Much of what's called open source these days has commercial aspirations. Often a lot of source code is developed and then published when it's quite far along, after the sheer volume of it provides barriers to entry. It's not really open source in the sense of leveraging the advantages that collaborative development in a community brings.
In academic settings competitive aspects of research likely produce the same sorts of issues, and in compsci there is likely commercial interests involved too. Come up with a clever new NPL technique for semantic searching and the VCs come out of the woodwork.
However I agree with the author and commenters that both code and data ought to be available to all, mainly because that's the only way to make progress. Research is hard, very hard, and building on the half-baked ideas, good and bad, is the only way progress is made.
Re: Let me see your papers, let me see your source
#5Open-sourcing is not enough. Honest researchers should also publish all data sets for peers to validate results. When I was in grad school, it was a disappointing fact that very few academics in the machine learning field did this.
There are some well known public data sets used for this purpose, such as those in the UCI Machine Learning repository. Unfortunately, not everyone is using them. And even if they do, it is often impossible to reproduce the results as pre-processing of the data is not described well enough in the paper, or because the authors add random components (such as costs) to the data without describing the distributions properly.
Publishing scripts for the complete workflow starting with the raw data and printing the table with the results in the end would be the best. But I've seen academics working in a way that is completely orhogonal to this - copying & pasting data to Excel or Matlab (or even re-typing them) and doing the analysis by hand in the GUI... I don't have any doubts they would be able to learn how to write the script, but I'm very sure they would put up heavy resistance to do so.
Re: Let me see your papers, let me see your source
#6People should be clear if they want to do something academic or commercial. If you want your code to be "secret sauce", then don't publish in academic journals, write a memo to investors or customers. If you want to do science, publish and be open and testable. Scientific publication serves to communicate and spread ideas, not to gain "reputation points" you can then exchange for credibility in the marketplace.
On a less polemic note: To address the reproducibility of computational experiments, Andrew Davison presented his Sumatra project at Euroscipy 2010 (http://www.euroscipy.org/talk/1960). I thought I'd throw this in here as the OP's gripes are a problem in all of modern science, not just computer science. It's also a reminder that making the code available is but one element of the problem of reproducibility/falsifiability in modern science (though probably the biggest one).
Re: Let me see your papers, let me see your source
#7Open-sourcing is not enough. Honest researchers should also publish all data sets for peers to validate results. When I was in grad school, it was a disappointing fact that very few academics in the machine learning field did this.
Exactly. Some data in my field (NLP) is available, however usually at high fees.
Coming back to the topic of source code. I think there are three additional reasons source code is often not published:
1. Some scientists (I am trying to avoid making overgeneralizations) are bad programmers. Sometimes just enough hacks are stacked to produce results, but the result is not something to be particularly proud of.
2. Rinse and repeat. It's often possible to get more out of a discovery by applying it to multiple datasets. If the source is published, others could be eating your lunch.
3. There is a contract that avoids publishing source code.
My PhD project is financed by the Dutch national science foundation. Fortunately, since software developed in my project adds to existing work under the LGPL (creating derivative works), my work is under the LGPL too. Copyleft can help you if (3) applies.
I try to follow the same strategy as the author: make software public on Github once a paper is accepted.
Re: Let me see your papers, let me see your source
#8Open-sourcing is not enough. Honest researchers should also publish all data sets for peers to validate results. When I was in grad school, it was a disappointing fact that very few academics in the machine learning field did this.
So true. I did some research on video retrieval a few years ago, where I wanted to make a user interface to see how we can benefit from all the cool segmenting and clustering techniques other people were researching. It turned out it was nearly impossible to get any data to build upon, I wasted hours and hours doing everything by hand. This severely impacts the ability to build upon the work of others.
Re: Let me see your papers, let me see your source
#9Open-sourcing is not enough. Honest researchers should also publish all data sets for peers to validate results. When I was in grad school, it was a disappointing fact that very few academics in the machine learning field did this.
But a lot of data cannot be opened for various reasons (privacy being a huge one) - see netflix prize 2 cancellation in the ML field.
There is also the issue of preventing competitor (other researchers here) to get a free ride on your work - getting data, preparing them is a huge part of the researcher's work in some fields.
Re: Let me see your papers, let me see your source
#10Reminds me of my own experience doing research in mobile robotics. It basically followed a cycle described pretty well by the the comic here: http://www.willowgarage.com/sites/default/files/blog/201004/...
From what I can recall of the six years I spent in academic research in the 90s the only thing that counted was publications - any effort into anything that wouldn't end in a publication was strongly discouraged.