Let me see your papers, let me see your source

It's not (exactly) my community, but SIGMOD has been trying to do this since 2008 with the "repeatability/workability committee." See this:

http://www.sigmod.org/sigmod-pods-conferences/mirrors/sigmod...

There's also an interesting FAQ about the repeatability requirements specifically here:

http://www.sigmod.org/sigmod-pods-conferences/mirrors/sigmod...

... as well as various follow-ups from that work, if you search on Google Scholar.

The issues as I understand them are:

1. Research code is often of poor quality, usually thousands of lines of unchecked output by a single graduate student. There are probably bugs, some of which may change the results. In the absence of code, results are assumed to be correct. It takes substantially longer to write good code, and hurts researcher "output" to do so. As a result, good quality code is actively discouraged.

2. Releasing code and/or data usually makes it substantially easier for others to duplicate or catch up to your research program, which is seen as a disadvantage if you are still in an area. It may lead to more citations, but that probably isn't enough for the effort. Other researchers also have an incentive to try to find bugs in the code/data, which they may overstate as they try to get their own work accepted.

3. The code and/or data itself my be copyrighted or have unclear distribution terms, for example, if you are doing experiments on a web crawl.

4. Actual production quality code that does something useful can be used as the basis of a startup or other venture, especially if the researcher is the only one who has and understands it. Furthermore, research groups can make money licensing their code to outside companies if they do not release it openly.

5. Many (most?) industrial papers involve code or data that cannot be released. Ultimately, in highly competitive conferences, it is hard to balance "unverifiable" papers written by industry with academia papers. A blanket ban on papers without code or data would remove a huge number of industry contributions, but an optional requirement for code or data mostly continues the status quo. Many of the most interesting recent papers (e.g., MapReduce) might not have been published with a code/data requirement.

Aside from legal issues, it seems to me the business proposition is the same as the open-source business proposition: you know the most about the system you've created, so you're in the best position to consult on it. If you want a startup, I think guys like Cloudera show that even if you give away what was traditionally thought of as the family jewels, you can still very effectively monetize. That's what the university should leverage.

Anyway, for most projects, you've already given the game away in the paper (or at least, should have done): the expensive thing was the idea. Reimplementation is cheap.

And for these other things, it might be good to see code a couple of times, but mostly at first, and to get up to speed in an area.

I think a problem is that, if you force all papers to include source code, due to the fact that science builds upon itself, you'd find that the average paper length for an area should climb a bit (going back down when a paradigm shift happens, because then the tricks stop working, but climbing all over again), and most of it would just be repeats of what's already there. Comparing to maths, it's like asking every paper proving a theorem to prove their lemmas, even very basic ones: sure, it'd help a novice understand what's going on, but it'll hinder progress more often than it'd help it. There's a place for introductory writing and a place for stand-in-the-shoulders-of-giants writing, and papers are mostly of the last sort.

For example, a couple of decades ago every paper that used a naive bayes classifier would derive the equations, describe feature selection, weighting, etc; today, most just say "I use a naive bayes classifier for this, that, and that" and move on. Likewise for SVMs---you don't want to see the full code for most papers that use that, since it's a mess of kernel caches and dual variables that mean nothing whatsoever to the problem at hand (but the algorithm won't work without it).