101–110 of 156 posts
Earlier quoted context omitted.
The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.
Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper. Are you sure you have thought this through?
Earlier quoted context omitted.
And who exactly is supposed to verify that code is reproducible for all the papers that get accepted?
The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.
Then the paper needs to be replicated a few times by independent research teams to be considered valid. This is the step we are probably missing in AI. Any published paper (and even unpublished, on arxiv) are considered valid and by the community but sometimes cannot be replicated.
Earlier quoted context omitted.
This sort of thing is aggravating to read. Frankly it comes off as really entitled. As researchers, the expectation is now that we not only have to do the research and write a paper like the good old days, but we have to release the code too. Okay, fine. But now that's not enough either -- the code has to be well-documented and clean. Ugh, alright, fine -- it's going to take me a few extra weeks of not doing research…
Pardon me, maybe I just misunderstood the whole idea of research but what good is it if it's not reproducible? I can understand it may be part of a meaningful personal journey for you, and I appreciate that. But if no one else can validate your research they're correct to discredit it and you. So what is the optimal outcome here? Should we hold you to a standard of reproducibility even if it is as minimal as, "actual…
Earlier quoted context omitted.
The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.
Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper. Are you sure you have thought this through?
This was a major topic at NIPS 2017, yet doesn't seem to be important enough to address.
Earlier quoted context omitted.
The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.
It's not the editor's job to replicate the results. Imagine the same requirements in medical research, this would be crazy. Editors accept a paper based on the quality of the paper and the claim. Then the paper needs to be replicated a few times by independent research teams to be considered valid. This is the step we are probably missing in AI. Any published paper (and even unpublished, on arxiv) are considered vali…
Speaking as someone who has read about 40 years of papers in compiler optimization, it's very interesting. In the early days, there were fairly exact algorithms that worked as described, and were implemented as described, but were pseudocoded in papers. Where the pseudocode differed from implementation, differences were described in great detail (IE they may say an array can be shared but isn't to make the pseudocode…
As someone who works on compiler optimisations, I'd love to get a link the Rice compiler's source code --- do you happen to have a link?
https://www.amazon.com/Engineering-Compiler-Keith-Cooper/dp/...
And the example of the paper with literate code:
https://www.cs.ucsb.edu/~ckrintz/papers/shared.ps.gz
Still it would be nice if somebody decides to get and also make available the papers or code for which the links are now dead, e.g. those linked from here:
Earlier quoted context omitted.
The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.
It's not the editor's job to replicate the results. Imagine the same requirements in medical research, this would be crazy. Editors accept a paper based on the quality of the paper and the claim. Then the paper needs to be replicated a few times by independent research teams to be considered valid. This is the step we are probably missing in AI. Any published paper (and even unpublished, on arxiv) are considered vali…
They obviously meant the reviewers. Not everyone is intimately familiar with academic terminology.
Earlier quoted context omitted.
Typically when I post code for my papers, I try to clean it up, because it's an ugly mess that is hard to understand (even myself sometimes). This clean up process might introduce bugs. I'm not going to thoroughly test everything, I have next paper deadline coming up and my adviser could not care less about my code. But, if you're interested in my research and having trouble reproducing my results, email me and I'll…
And don't you think it would have been more efficient to write it down clean at first with good comments and code practice ? Those good practices are here for a reason and not just for annoying you for the sack of "beauty". You would save more time for yourself when you wanna expand the paper further or just reuse some part of it, have better reputation for your work, and gain some citation because people will have t…
The software types love to sht on researchers for their poor quality code. Yet a few threads over, there’s always a discussion about how more tests don’t make it onto the kanban board (or whatever you guys are using these days) when crunch time approaches. It’s not just us.
Earlier quoted context omitted.
The editors, previous to accepting. If it doesn't run out of the box and reproduce the result, it doesn't get published.
Some code might require 8 GPUs and a week to verify a single result. There could be a dozen results to be verified in a paper. Are you sure you have thought this through?
Earlier quoted context omitted.
One explained to me here that the problem is they're mainly rewarded by funding agencies for how many papers they publish. That's why quality went down in general. Then, since funding agencies aren't rewarding code, they're either not encouraged to get it right or actively discouraged from doing it to focus on next paper. This sad situation is both bad for open source and science. Bad for science since inaccurate res…
Then some other academic can get credit for publishing a rigorous paper describing the actual algorithm, explaining why the original paper is wrong (without speculating on the motive, just the facts), and why the approach taken in the code is in fact better. Bonus points for describing a further improvements, with working code.