As is usual for Stephen Wolfram, he has a point, and then blunts it by trying to own the whole thing. Edit: to expand, part of the answer to his question, why don't more people do this, is that it requires his expensive proprietary software. Scientific papers are (nominally at least) a commons.
Redesigning the Scientific Paper
31–40 of 110 posts
Re: Redesigning the Scientific Paper
#32This title is horrible hyperbole. Science is more than just machine learning. Hell, even if we just constrain ourselves to "computer science" probably half of it is just math, for which the scientific paper is definitely not "obsolete" nor even deficient in any way. But outside of computer science you need laboratories to replicate experiments. Scientific papers are perfectly fine vehicles to record the necessary inf…
OK, we'll hypobolize the title a bit.
But it would be better to react to the substance of the article, which is more interesting.
Re: Redesigning the Scientific Paper
#33I used to work as a software developer for a research institute. I wanted to open source our research code and tools, and the department head was in favour of it because it would raise the profile of the research unit.
There were two forces working against us. First many of the grants came from governments, and a stipulation was that we would devote some resources to helping startups commercialise the output of the research. Some felt that open sourcing would remove the need for the startups to work directly with them to integrate the algorithms, and that this would hurt future grant applications by making the research look ineffective.
The main opposition though came from PhD and Postdoc students. Most didn't want anything related to their work open sourced. They believed that it would make it easy for others to pick up from where they were and render their next paper unpublishable by beating them to the punch.
Sadly I think there was some truth to both claims. Papers are the currency of academics, and all metrics for grants and careers hinge off it. It hinders cooperation and fosters a cynical environment of trying to game the metrics to secure a future in academics.
I don't know how else you should measure academics performance, but until those incentives change the journal paper in its current form is going nowhere.
Re: Redesigning the Scientific Paper
#34We need GitHub for science. But that's not enough. It needs to be combined with a mechanism for peer-review and publishing that funding agencies will find acceptable--that's the key.
It's not only about funding agencies, though. Researchers need to be trained in the use of a GitHub like system - as a former graduate student, I knew NO ONE using Github. Furthermore, after working in scientific publishing where it was MS Office or bust, I doubt you will convince Elsevier and the publishing giants that GitHub for Science is the way to go. How do these huge companies make money if scientists are shar…
When did you do your PhD? When we collaborated for papers we used Subversion (at the start of my PhD) & git (towards the end) both for code and papers (in latex).
Re: Redesigning the Scientific Paper
#35We need GitHub for science. But that's not enough. It needs to be combined with a mechanism for peer-review and publishing that funding agencies will find acceptable--that's the key.
I would really like the Open Science Framework to become just that (with other tools like OpenML as needed). But it requires people to actually work on it to happen...
Re: Redesigning the Scientific Paper
#36> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…
Have you seen OpenML? There are solutions for this, and I think most people would agree they are useful, it's just the change/adoption/standardization cost is high as always.
Re: Redesigning the Scientific Paper
#37as a practical matter, papers will remain relevant as long as they are the metric by which grant applications and tenure decisions are made. as a philosophical matter, for computation heavy fields, i would love to see literate programming tools become de rigeur in the peer-reviewed distribution of results. In some fields (AI) this basically happens already — the blog post with code snippets and a link to arxiv at the…
Papers (and PDFs) are relevant because they are easy to organize and archive, essentially in perpetuity. Source code is too, so nothing wrong with a "Github for Science". Notebooks, blogs, or interactive dashboards, on the other hand, are an amazing tool both for research and for communication, but they are far more ephemeral than a paper. They need a large overhead to keep them running that cannot be sustained over…
Those are good points. Journals have to care about this, too, these days -- supplementary information now routinely includes videos (hope you have the right codecs!), word documents, audio, and full or redacted data sets.
I think what appeals to me about literate programming style is that it encourages a return to a more clear and expository style of writing, which has been squeezed out of scientific writing in journals over the years. I don't care what instantiation is required to produce a more uniformly clear and cogent document, I just care that it happens.
Re: Redesigning the Scientific Paper
#38Earlier quoted context omitted.
I've had a paper peer reviewed. It was ultimately rejected but I can't help but suspect that by making all my code publicly available, I hurt my chances of publication. The reviewers comments were about my coding style, my choice of build tool (I didn't use make, but something else which is just as easy to use), the choice of C vs C++... It's like best practices for computer security -- always strive to minimize the…
> It's like best practices for computer security -- always strive to minimize the attack surface. I suspect that's also why some papers are unnecesarely verbose and describe simple things as complicated as possible. Can't criticize something that can't be understood.
Then why submit it for peer review at all?
Re: Redesigning the Scientific Paper
#39> How to integrate billions of base pairs of genomic data, and 10 times that amount of proteomic data, and historical patient data, and the results of pharmacological screens into a coherent account of how somebody got sick and what to do to make them better? How to make actionable an endless stream of new temperature and precipitation data, and oceanographic and volcanic and seismic data? How to build, and make sense of, a neuron-by-neuron map of a thinking brain? Equipping scientists with computational notebooks, or some evolved form of them, might bring their minds to a level with problems now out of reach.
The article seems to conflate the praxis of science with the archival of it. Scientists do all of the above on gigantic clusters, not on an IPython/Mathematica notebook. The purpose of publishing papers, on the other hand, is adding to the archival of knowledge, and they can be easily rendered in a laptop with LaTeX.
And they are excellent at archival, by the way. You can see papers from the 19th century still being cited. On the other hand I have had issues running a Mathematica notebook from a few releases back -- and I seriously doubt one will be able to read any of my Mathematica notebooks 150 years from now. The same with the nifty web-based redesign of the Nature paper that is mentioned: I bet the original Nature article will be readable 150 years from now, whereas I doubt the web version will last 20.
Re: Redesigning the Scientific Paper
#40> These programs tend to be both so sloppily written and so central to the results that it’s contributed to a replication crisis, or put another way, a failure of the paper to perform its most basic task: to report what you’ve actually discovered, clearly enough that someone else can discover it for themselves. This is the crux of the of the problem IMHO - at least for the fields I study (AI/ML). Replicating the resu…
> there is a disproportionate focus on paper quality over "project quality"
One problem is bitrot. Stuff that runs now is not guaranteed to work in 1 or 2 years, let alone 10 years.
Even more so when it runs on fancy hardware, like GPUs.