Let me see your papers, let me see your source • Hacker News Board

Re: Let me see your papers, let me see your source

#11

Post by alextp » Sat, Jul 24, 2010, 3:23 PM UTC

I agree with the feeling, but a lot of the time, scientific software is just not ready for real usage. My software at least is completely useless most of the time (you have to open it in the interpreter and type the right incantations, and the data has to be in a non-trivial massaged way, filesystem paths are hardcoded everywhere, etc). It takes some effort to turn a works-for-me research tool into something releaseable, and this effort might even hinder future research, since the fluidity of the hacky code is sometimes necessary for further tweaking. Also, there are usually dozens of dependencies, some of them patched to work with that software, with different incompatible licenses, etc.

I think, however, that most journals should at least ask for a shell script that downloads the code and data, runs the experiments and regenerates the graphs and tables as seen in the paper. This is not always practical (a lot of papers deal with over a terabyte of data, for example), but it is so more often than not. At least for my papers this is a nearly-attained goal.

Re: Let me see your papers, let me see your source

#12

Post by miloshh » Sat, Jul 24, 2010, 3:25 PM UTC

Source code publication along with the paper is a great idea in theory, but there are several problems with it in practice.

The reason CS researchers do not usually publish their code has nothing to do with dishonesty - nobody is trying to hide their code because it does not really work, or anything like that. It's not even that people are worried of scooping, though that sometimes happens.

The main problem is that any time spent on cleaning up the code, packaging examples, writing instructions, answering bug complaints, etc. is time not spent on things that matter in academia - doing research, presenting it, and teaching students.

It might help if some conferences required source code submissions - but people might just submit to different conferences instead. The only real solution would be if funding agencies like NSF required that any projects funded through them have to release source code. This makes sense from a taxpayer's point of view, and would make the extra work acceptable (since everyone would have to do it).

Re: Let me see your papers, let me see your source

#13

Post by miloshh » Sat, Jul 24, 2010, 3:35 PM UTC

alextp wrote: post #11

I agree with the feeling, but a lot of the time, scientific software is just not ready for real usage. My software at least is completely useless most of the time (you have to open it in the interpreter and type the right incantations, and the data has to be in a non-trivial massaged way, filesystem paths are hardcoded everywhere, etc). It takes some effort to turn a works-for-me research tool into something releasea…

I was just typing my comment while you posted yours. I absolutely agree with this. Especially the "fluidity of the hacky code" - the environment that makes cool research happen is often the opposite of clean software engineering.

Re: Let me see your papers, let me see your source

#14

Post by RaphaelWimmer » Sat, Jul 24, 2010, 3:45 PM UTC

cdavid wrote: post #9

antichaos wrote: post #3

Open-sourcing is not enough. Honest researchers should also publish all data sets for peers to validate results. When I was in grad school, it was a disappointing fact that very few academics in the machine learning field did this.

But a lot of data cannot be opened for various reasons (privacy being a huge one) - see netflix prize 2 cancellation in the ML field. There is also the issue of preventing competitor (other researchers here) to get a free ride on your work - getting data, preparing them is a huge part of the researcher's work in some fields.

This "getting a free ride on your work" is also called "standing on the shoulders of giants".

Re: Let me see your papers, let me see your source

#15

Post by robryan » Sat, Jul 24, 2010, 3:47 PM UTC

I see some of the comments here talk about the code being useless it terms of using it the same way the author of the paper has. Personally for a lot of the papers I have read in NLP based stuff I'd feel more comfortable just having some source to look through to give a better idea of how it was done. Some papers are quiet abstract about their method and aren't really helpful past giving you basic ideas without some hard code to let you fully understand what they have done.

In contrasting it to maths it would see stupid for a maths paper to describe a new maths proof without putting it forward in mathematical notation in the paper.

Re: Let me see your papers, let me see your source

#16

Post by Lewisham » Sat, Jul 24, 2010, 5:02 PM UTC

alextp wrote: post #11

I agree with the feeling, but a lot of the time, scientific software is just not ready for real usage. My software at least is completely useless most of the time (you have to open it in the interpreter and type the right incantations, and the data has to be in a non-trivial massaged way, filesystem paths are hardcoded everywhere, etc). It takes some effort to turn a works-for-me research tool into something releasea…

(Disclaimer: Author of the OP)

I absolutely understand your feelings on hacky code. Every academic produces hacky code, there are precious few who don't. I myself, when I started, did not want to release my code for the same reason.

However, once I began to realize that we were all on the same boat of HMS Hacked Together, that feeling began to dissipate. My advisor calls it "research code", and it's fine, because as academics, we're all used to it!

That's why I usually just ask for source. I assume the build won't execute on my Mac, and that's OK. I'm not really interested in running the tool, but explicitly finding out how you solved the problem.

Re: Let me see your papers, let me see your source

#17

Post by Lewisham » Sat, Jul 24, 2010, 5:05 PM UTC

miloshh wrote: post #12

Source code publication along with the paper is a great idea in theory, but there are several problems with it in practice. The reason CS researchers do not usually publish their code has nothing to do with dishonesty - nobody is trying to hide their code because it does not really work, or anything like that. It's not even that people are worried of scooping, though that sometimes happens. The main problem is that a…

(Disclaimer: I am author of the OP)

The NSF is a great point.

One of the things that bugs me about this, that I didn't go into in the post for brevity's sake, is that a lot research is funded by some government institution under the banner of public interest. If you are paid to create something, and then you lock it away for whatever reason, that's not in the public interest. Worse still, more money has to be spent for someone else to re-implement the exact same thing if they liked it!

I hadn't thought this out to the logical conclusion of having the funding body also ask for the code to be released, but I think it's a great idea.

Re: Let me see your papers, let me see your source

#18

Post by Lewisham » Sat, Jul 24, 2010, 5:11 PM UTC

RaphaelWimmer wrote: post #14

cdavid wrote: post #9

Earlier quoted context omitted.

But a lot of data cannot be opened for various reasons (privacy being a huge one) - see netflix prize 2 cancellation in the ML field. There is also the issue of preventing competitor (other researchers here) to get a free ride on your work - getting data, preparing them is a huge part of the researcher's work in some fields.

This "getting a free ride on your work" is also called "standing on the shoulders of giants".

(Disclaimer: I'm the author of the OP).

Yes, but the person you are replying to is right to note the competitive aspects of research. A lot of people might say "well, this project is on-going, and I don't want people scooping/stealing it from me." It's a sad thing, but most research labs are in a a competitive relationship with other ones, and a citation is less useful in those oh-so-important tenure reviews than a publication. I wish it was more the "standing on the shoulders of giants"!

That's partly why I didn't call on academics themselves to release code, but for some sort of authoritative institution instead, to level the field for everyone. That should remove the competitive aspects (I understand "that should" is a very naïve outlook ;) )

Re: Let me see your papers, let me see your source

#19

Post by ewjordan » Sat, Jul 24, 2010, 5:34 PM UTC

cdavid wrote: post #9

antichaos wrote: post #3

Open-sourcing is not enough. Honest researchers should also publish all data sets for peers to validate results. When I was in grad school, it was a disappointing fact that very few academics in the machine learning field did this.

But a lot of data cannot be opened for various reasons (privacy being a huge one) - see netflix prize 2 cancellation in the ML field. There is also the issue of preventing competitor (other researchers here) to get a free ride on your work - getting data, preparing them is a huge part of the researcher's work in some fields.

Academics in some fields build up entire careers worth of work that all builds upon their "special sauce" super-secret dataset or software, which they've built up over many years, and while there might be enough information in principle in their papers to reproduce the data or results, it would be a thesis level project to actually do so.

There's a story (unfortunately I forget where, perhaps someone else can jog my memory?) about a string theory PhD student who, after getting annoyed wasting months on a set of hundreds of straightforward but tedious and time consuming calculations, decided to just do it all once and for all and spend the time to put together a table of results for all of them. Of course, this helped his future work immensely.

When he went to his PhD advisor to ask what he thought about publishing that table, the guy looked at him like he was crazy. Again, I don't remember the quote, but it was along the lines of "What you have there will give you a 1000% speed advantage pushing out papers in this field compared to your peers - you'd have to be crazy to share that sort of competitive advantage with everyone else when you could keep it to yourself, this is your golden ticket!"

There is also the issue of preventing competitor (other researchers here) to get a free ride on your work - getting data, preparing them is a huge part of the researcher's work in some fields.

Which brings to light very clearly the source of the problem - the ideal of academia is to advance the overall state of knowledge as fast as possible, but once you start using the word "competitor" in a serious way that actually has bearing on whether you publish something useful or not, that ideal has been perverted.

Obviously it's the "publish or perish" mindset that causes this, and I absolutely understand why people would be tempted to see their supposed colleagues as competitors instead of collaborators (in the general sense, when they're not actively collaborating on a paper); it's one of the main reasons I decided not to go into academia, in fact, I saw too much political bullshit spewing about even in the harder fields like physics and math. I have no idea how to solve any of this, but it's a serious breakdown in the system, and I suspect it (publish-or-perish, not just information hiding) hinders the long term progression of academic knowledge in some of these fields by a large amount, not in the least because it rewards herd-like behavior and punishes exploration. That's another can of worms for another day, though...

Re: Let me see your papers, let me see your source

#20

Post by _delirium » Sat, Jul 24, 2010, 5:55 PM UTC

Lewisham wrote: post #16

alextp wrote: post #11

I agree with the feeling, but a lot of the time, scientific software is just not ready for real usage. My software at least is completely useless most of the time (you have to open it in the interpreter and type the right incantations, and the data has to be in a non-trivial massaged way, filesystem paths are hardcoded everywhere, etc). It takes some effort to turn a works-for-me research tool into something releasea…

(Disclaimer: Author of the OP) I absolutely understand your feelings on hacky code. Every academic produces hacky code, there are precious few who don't. I myself, when I started, did not want to release my code for the same reason. However, once I began to realize that we were all on the same boat of HMS Hacked Together, that feeling began to dissipate. My advisor calls it "research code", and it's fine, because as…

(Disclaimer: I work in your lab. ;-))

I've asked people for code a few times, but my experience after getting it is actually that I don't really ask for it anymore, because I've never found it to help me. What I really want in most of the cases is a clear enough English writeup, perhaps with pseudocode, so that I can understand how they solved their problems, and ideally reimplement it myself. At least, that's the case if it's at a scale where that's feasible to reimplement; if they built something absolutely gigantic then it might be another story, but then their megabytes of messy research code I can't grok aren't very useful to me either, and I have no real choice but to wait for the cleaned-up release.

In short, I think "can this be reimplemented by a third party from the published literature?" is a better test for reproducibility than .tar.gzs are. And there's certainly a ways to go on that front, not least because in areas where 6-to-8-page conference papers are the norm, even well-meaning authors can't include enough details, and most don't get around to writing the detail-laden tech report version. But I guess I find code mostly useless for that purpose; it might as well be an asm dump for all the good I usually get out of it.