The Future of Peer Review

The quality of peer review is only as good as the editorial board and reviewers make it -- often this is a lower standard than we expect. This article lists several extreme examples of poor practices by a (previously-)prominent publisher:

http://blog.mathunion.org/journals/?no_cache=1&tx_t3blog...

The key to shifting academic quality control away from the present archaic model is through author's motivation. The replacement needs to be a system that improves and filters incoming papers, and at the same time earns at least as much prestige for the authors who participate.

One idea is something like a combination hacker news and stackoverflow with academic articles in place of news articles / technical questions as the object of discussion. Papers can be changed in response to comments. High scores for either a paper or an author on such a site could replace the prestige granted by acceptance into a top-tier journal. Points are awarded for reviews, and researchers are respected for their scores (closer to tenure), so there is more positive motivation to review than in today's system (which is: it's rude not to review when asked).

I think that alone could work. Some academics might balk at a lack of editorial boards. Fine, let's allow some editors in the mix. An editorial board is an aggregator. They choose what content fits their filter, and may open a discussion with an author on improvements, possibly bringing in reviewers. In the end, readers will see a prestigious badge of the board's approval on high-quality articles, and there could be a page or rss feed to browse the articles meeting this standard. This can all exist on top of the hacker news / stackoverflow system, which itself could be built on top of an arXiv-like system.

Several of these ideas are motivated by thoughts from the blogs of Tim Gowers and Michael Nielsen.

I agree that discovery and credit/prestige are the two key things to get right. I'm sure that the kinds of systems you're talking about, voting systems, will be part of the research discovery/credit process in the future. It works for web content, and also for very esoteric content on the web (MathOverflow etc), so it should work for research papers too.

Trying to use general web search is often far too noisy. Often times I find interesting papers on page two, three, or four of google scholar. In a general web search these same papers might be on a much later page, or perhaps buried so far that they don't come up at all.

As others have said, there is also much more to peer review than discovery. Peer review is additionally intended to help authors improve work that isn't quite up to standards. Academics do peer review for free because they recognize the value of it and because they are asked directly by the editors to do it. In your model, what would be the incentive to look for new papers and give reviews? How would you handle old versions of papers with mistakes or papers that are of insufficient quality? The peer review process currently filters these intermediate stages of paper writing. In your proposed model, you would potentially have many versions of the same paper that a user would then have to filter through, in addition to a very great many papers that are of very low standard.

The quality of content on the web and its curation is a very poor standard to compare academic literature to. Many users are happy enough to be able to filter out obvious spam pages let alone judge quality to the level needed in academics.

One of the drawbacks of the existing peer review process is that academics don't get credit for their reviews, nor do the reviews see the light of day for others to benefit from. I expect that there would be significantly more discussion, and reviews, of papers in the future if there was a credit system that allowed people to get credit for reviews and comments they made of papers. I think that credit system is possible and will be built.

There is an interesting question regarding the immutability of content. Right now, once you publish a paper, you can't edit it, or delete it. It's an immutable piece of content. Before the web was established, there was a line of thinking, developed by Ted Nelson, according to which the internet should evolve like that, and that a link should always work: once some content is posted, it can never be taken down. Most people are probably glad that the web developed along the lines of Tim Berners Lee's thinking, rather than Ted Nelson's, and that they are now free to edit and delete content they have posted. I think similarly people would appreciate being able to update a paper in response to a comment they have received. The author is better off, and so are subsequent readers, as they find themselves reading a more evolved and advanced paper.