51–60 of 175 posts
I think this is a great idea theoretically, but in reality for most papers I don't want to see the data/underlying code. While it would be great to publish data/code with the paper (in the field I've worked on the most, astronomy, most data is already published with the paper anyways), I don't want/need to look through a notebook with the underlying code of the paper in order to just read the intro/conclusions (and m…
As a practicing scientist, I firmly believe the world would be much better off if we simply published version-controlled Jupyter notebooks on a free site, such as GitHub or ArXiv.
It's sort of code, but more convenient for just getting work done. You can pass around the whole thing (data + code). No need to learn software development skills or set up a development environment to replicate results; it runs everywhere. Plenty of power to do advanced things. Already a standard in research.
Earlier quoted context omitted.
Would the idea be valuable without the code? Yes.
If I've learned anything in my career it is that no, ideas are not valuable. There are vastly more bad ideas than good ideas. What makes an idea valuable is validation. Papers aren't to present ideas: papers are to present ideas that have been validated. We proposed an idea, we went and ran some experiments or gathered data some other way, and we concluded the idea was valid (or not valid). The point of this discussi…
Likely the tech stack you use is built on a tower of ‘just idea’ papers.
I feel like the website paperswithcode.com addresses this very well, especially with their feature "quick start in Colab". For example, here's the top paper on the website as of now: https://paperswithcode.com/paper/towards-real-world-blind-fa... . Instead of going through the process of cloning a repo, initializing a fresh Anaconda environment from scratch, reading through nebulous, haphazard documentation about how…
It's really disappointing that technical societies like the ACM and IEEE haven't done this already. For many journals and conferences there isn't even a way to submit the code or other digital artifacts with the PDF. A few have badging for whether digital artifacts are provided and whether the results have been reproduced or repeated by others - steps in the right direction at least. As much as I intensely dislike th…
Earlier quoted context omitted.
Yes but chances are it only appears to work because the analysis code has bugs. So first i want to check the code and that it works before i put effort into understanding the idea.
So let me clarify. You're saying that the majority of CS papers only appear to work because the analysis code has bugs? And that checking the code (presumably also the analysis code) is easier than "understanding the idea"? Neither of those ring true to me, but your mileage may vary.
As a practicing scientist, I firmly believe the world would be much better off if we simply published version-controlled Jupyter notebooks on a free site, such as GitHub or ArXiv.
> As a practicing scientist > version-controlled Jupyter notebooks That's awfully field specific. It probably wouldn't work for most of STEM. Even for ML I shudder to imagine trying to make sense of the inevitable monstrosities. Writing a paper is part of the thinking process. It forces the author to sit down and work through things in an orderly manner and they're still often difficult to read. I'm definitely in fav…
As a former academian: Papers are difficult to read primarily because the academic community does not value making them easier to read - no other reason. You may hear things like "papers should be written for other experts", but even that doesn't hold up to scrutiny.
They typically spend 99% of their research time on the actual research, and less than 1% on writing the paper. They definitely can afford the time and energy to make the papers easier to read, but game theory holds sway: Why should a particular researcher use his/her time to do it, when his/her peers will not appreciate it? It's purely an internal, cultural problem. There are no external constraints leading to this.
I've seen referees send papers back saying they contained too much explanation, and suggest leaving out most of the details - just include the big picture methodology and show the results. I can guarantee most who will read the paper would not be able to reproduce those details, if they ever want to. Likewise, I've found papers where I couldn't reproduce the results, because the results were wrong - but since including a derivation of your final expressions is discouraged, no referee caught the errors.
Earlier quoted context omitted.
I think this almost every time I read the paper. It’s like Linus’ “show me the code.” I just want papers now to “show me the data and the code.” And include a discussion about why these results are important. I think it’s a great time for the scientific community to improve transparency on these fronts. Sincerely, someone who reads a lot of research but contributes none because I’m an amateur. Edit: when I say data,…
> I just want papers now to “show me the data and the code.” But the code is secondary to the idea. The idea and the discussion around how it was arrived at and what it means is the key thing. The code is just there to implement it. You could code the same idea ten different ways.
If anything, I'd like to see more focus on giving evidence of generality of a result, vs just sharing everything needed to get back the same specific result
As an academically employed scientist of 20 years, the notion that scientific communication suddenly needs better standards puzzles me. The core research curriculum of nearly every scientific field I’ve seen, STEM or otherwise, is that the data needed for replication are non-negotiable. A paper that doesn’t include it would be table rejected by any editor. Or one would hope. This is taught at the UNDERgraduate level,…
This may vary based on discipline, but in both the subdisciplines of experimental and theoretical physics I was involved in: No - very few will provide the data/derivation. My professors were very open about this: They don't want to lose their competitive edge. Almost no experimentalist I knew could take papers from his/her field and reproduce the results, because the papers lacked enough detail to do so. They would mention a technique, but there are lots and lots of nuances involved when building equipment to carry out the technique[1], and these are intentionally excluded. It's unlikely you'll be able to build the equipment the same way the original authors would.
[1] Most experimental physics involves building your own equipment, or at the least modifying existing equipment.
Earlier quoted context omitted.
Yes. We should record all of this, and turn them into easily browsable graphs/hypertext to easily assemble sets of papers to read/look into. At the very least things like 'background reading', 'further reading', 'supporting evidence' and 'addressed arguments' would be useful. 'We' meaning the librarians and archivists. You guys actually researching have more than enough to do.
Actually I think that's an intriguing idea for how to improve citations. Instead of a single citation, have multiple citations that include a region of text as well as basic categorization of how the citation is being used in that instance. I'm not sure if it would prove feasible in practice. It seems like it would aid the writing process in some cases by helping the author keep track of details. But in other cases m…
Definitely don't want to encourage papers to take even LONGER.
There is little effort in making your results understandable and easy to replicate. Academia values paper production, which requires convincing peer reviewers that your results are not trivial and are worth publishing. Contrary to what the essay states, I don't think many scientists today think their research is "incremental". In fact, this word is used in many places as a derogatory term to indicate certain result doesn't contain enough novelty to deserve publication. Researchers are more incentivized to make their constructions and results as complicated and less accessible as possible.
This is not just a theory, this is something I've seen over and over throughout the years.