The Future of Peer Review

For an example of lousy peer review, I posted this yesterday on LessWrong:

A new science journal recently published a seriously crackpot paper, this has the abstract a link for the pdf. I first heard about it from Derek Lowe, who has also written two follow-up posts. The first has a couple of links discussing how news of the paper spread, while the second includes a link to the journal making excuses for why they published it.

Moreover, members of the Editorial Board have objected to these papers; some have resigned, and others have questioned the scientific validity of the contributions. In response I want to first state some basic facts regarding all publications in this journal. All papers are peer-reviewed, although it is often difficult to obtain expert reviewers for some of the interdisciplinary topics covered by this journal. I feel obliged to stress that although we will strive to guarantee the scientific standard of the papers published in this journal, all the responsibility for the ideas contained in the published articles rests entirely on their authors.

I included the links to all of Derek Lowe's posts because they have other interesting links, including in the comments.

The permalink is here http://lesswrong.com/lw/9p9/open_thread_february_114_2012/5t... , so you can check out the links I included.

I'm all for new publishing models in academia, but this post doesn't address the fundamental difference between the web in general and academic papers: expert curation.

The web's "curation" works (to whatever degree it does) because there are thousands, if not millions of people who can rate the quality of most articles/essays/websites. But in many scientific fields, there are often only a very small number of people who would be qualified to review a particular paper (usually well under a hundred, sometimes as few as a handful).

And make no mistake, this is a numbers game -- there's the well known 100/10/1 rule for content websites (for every 100 viewers, there are 10 who rate content, and only 1 who creates content). The exact numbers vary quite a bit from website to website, but the point is that it's orders of magnitude difference. That's all fine if your initial pool of people is huge, but if you start with 20 qualified people...you see the issue.

Another related point not addressed is that peer review is fairly demanding work. You're not just saying "accept" or "reject", but you're obliged to write a page or two suggesting the strong and weak points of the paper, things to improve, etc. You need some mechanism for enforcing this (even if it's via social norms), otherwise review quality deteriorates quite severely, which in turn affects paper quality.

In lieu of this article, I'd recommend seeing lectures by (or essays by) Michael Nielsen, who's thought about this problem in much more detail and (I believe) has a much better understanding of the way things work.

http://michaelnielsen.org/blog/essays/

Am I the only one that finds it ironic that TechCrunch is writing about peer review?

Peer review often works the worst when a journal asks someone who's not that qualified to peer review a paper. See the comment below from billswift about an unbelievable case of peer review going wrong, where the reviewers weren't in the research area of the target article (you need to follow the links to get the full picture).

Something doesn't have to be heavily cited for it to start getting traction, just as a website, or blog post, doesn't have to get heavily linked for it to get some traction. A friend might find the paper and share it via Twitter, their blog, or Academia.edu; this is how the social review process works on Twitter and Facebook. It might sound slow: you might wonder 'won't it take ages for a paper to get traction like that?'. But in reality, when you have social platforms like Facebook and Twitter, content can be surfaced, and whip around the world, at incredible speeds. The journal peer review process, by contrast, is several orders of magnitude slower.

The web is known for having a long tail of content: lots of content that appeals to certain niches of people. It is part of the magic of the web's discovery channels that this long tail content gets routed to the niches who care about it - via search, Facebook etc. The fact that academics overwhelmingly use the web now for research discovery is testament to how good a job the web's discovery channels are doing at surfacing good content in highly niche areas.

Wybo - thanks for these links. I totally agree that breaking the feedback loop you mentioned, that binds academics to the current journal system, is key. I think the central things that needs to happen are: building an alternative discovery system (already happening with the web), and building an alternative credit system (happening with citation counts, and other metrics, but is lagging behind the growth in alternative discovery systems, and needs to be accelerated).