Earlier quoted context omitted.
Well, I wonder if this could force people to start publishing their code so others can replicate the results. This would help greatly with the kinds of issues that are being discussed: - It would be much harder to hide makeup results and cherry-picked seeds - Useless research would not be as easy to hide - The real-world impact of the research would be more valued. - Researchers would be forced to recheck their code…
I'm active in the security and privacy field, and some conferences have started "artifact reviews" [1][2], where authors can optionally submit the data and code that go along with their work. It's definitely an improvement, but the devil is in the details. Should these be mandatory? If they're mandatory, then what quality standards should artifacts meet? Who's ensuring those standards are followed? How should proprie…
So cool. I wish there were more steps in this direction.
> Should these be mandatory?
As an outsider, I would say no. However that should have a detrimental effect on the ability to get published if you are from an academic institution. I can totally see a gray area when it comes to non-academic publishers.
> There's also the question of what code should be made public
Shouldn't it be as much as possible? The way I see it is that the easier to replicate a paper, the more people can focus on its true utility. That is, by abstracting ourselves away over the details of the paper, what does this really contribute? From my point of view, publishing as much as possible maximizes the real-world impact of the research.