Retire This Idea: Scientific Knowledge Structured as “Literature”

While not essential to the argument, it uses software development as an analogy for scientific publications. Unfortunately, that analogy isn't correct.

> Just as the software industry has moved from a "waterfall" process to an "agile" process—from monolithic releases shipped from warehouses of mass-produced disks to over-the-air differential updates—so must academic publishing move from its current read-only model and embrace a process as dynamic, up-to-date, and collaborative as science itself.

"Agile" is one name for a long line of software processes, which includes XP, RAD, and Unified Process. Iterative development stretches back to the dawn of software engineering. In modern practice, "agile" seems used for anything which is even a touch more flexible than the strictest of waterfall. Take your process, add a standup meeting or story cards, and poof, you're agile.

Nor is "agile" development tied so closely to 'over-the-air differential updates'. The development methodology has very little to do with deployment. Perhaps the easiest example is to look at the various language implementations (Python, Ruby, gcc, etc.). They don't have auto-update. So I guess the question is, are they agile?

> But academic literature makes no distinction between citations merely considered significant and ones additionally considered true. What academic literature needs goes deeper than the view of citations as kudos and shout-outs. It needs what software engineers have used for decades: dependency management.

> A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing.

I have no idea of how to make this work. I give as an example the Kabsch algorithm algorithm for RMSD minimization of two molecular structures (see http://en.wikipedia.org/wiki/Kabsch_algorithm ). The original paper contained an error that sometimes resulted in a sign error. This was fixed in a subsequent paper.

Most cite the first paper, because it introduced the core concepts. Some cite the first paper but actually implement the correct solution from the second paper without even realizing there was a problem, because they combined the principles with their own mathematical understanding rather than duplicating the exact steps in the paper. Still others cite both papers.

How does this hypothetical dependency graph incorporate those details?

That paper is a simple one, because it really is a single algorithm for a single goal. Many papers introduce multiple concepts, solutions, or observations. It can well be that one of them was wrong while the others were correct, interesting, and influential. For example, in one paper I read the authors did not include a test case because, they said, the source material was invalid. I investigated, and discovered that it was due to a transcription error by the authors of the paper, and not a problem in the source material.

The addition of that one data point would have no material impact on the overall paper. But it's still wrong.

How does his hypothetical dependency graph capture the multitude of disprovable points in the average paper? How does it capture what the "pillars" are?

Bear in mind that there's been over 50 years of attempts to do this, so I think there's no easy answer.

> How does his hypothetical dependency graph capture the multitude of disprovable points in the average paper? How does it capture what the "pillars" are?

How about bug trackers? Papers that are continuously revised as they're published? Scientific prestige based on how often your papers are cited and how active their bug trackers are?

That's how we do it in programming. I think the analogies could be stretched to scientific publishing.

The title made think this article was going to be about how to write science. Writing of all kind benefits if you incorporate elements of a story.

Cool ideas. We will probably end up with a better world if someone implements them, but I would argue that these "agile" things are much less important in scientific publications than in software development. Science and technology are different.

Software artifacts are huge and uncontrollable. Today it is very rare for a software developer to know all the ins and outs of a software he/she develops, along with all its dependencies. There is simply way too much code, and it is impossible to control them manually.

On the other hand, scientific papers tend to contain very few core ideas. It is still reasonable today for a researcher to understand everything he/she publishes, along with every idea in all the papers he/she cites. Once you get the core idea, you get the whole paper. They merely act as mediums for conveying ideas and there are not that many of them. Often you can just "manage" them with your own brain.

Anyway, I resonate with the first few paragraphs about retraction a lot. It is frustrating to read a paper for a whole day, find out that it is wrong, and that someone already refuted it in another paper :-/

Yes. Scientific knowledge structured as literature means scientific knowledge as subject fields which basically means a lot of lacking knowledge between these fields.

It's not that we should not have specific categories but rather that we should have a lot more than we have right now.

"Removing literature" as a framework for knowledge will mean the flourishing of knowledge that can only be created by combining several fields.