Why the world of scientific research needs to be disrupted

In their 1970 Annual review chapter on Memory and Verbal learning, Tulving and madigan reported that they had independently rated 540 published articles in terms of their "contribution to knowledge". With "remarkable agreement", they found that they had sorted two-thirds of the articles into a category labelled "utterly inconsequential". "The primary function these papers serve is to give something to do to people who count papers instead of reading them. Future research and understanding of verbal learning and memory would not be affected at all if none of the papers in this category seen the light of day"

About 25 percent of the articles were classified as: "run-of-the-mill .. these articles also do not add anything really new to knowledge... [such articles] make one wish that at least some writers, faced with the decision of whether to publish or perish, should have seriously considered the latter alternative"

Only about 10% of the entire set of published papers received the modest compliment of being classified as "worthwhile". Given that memory and verbal learning was then a popular and relatively 'hard' area of psychological research, attracting some of the brightest students, this is a devastating assessment of the end product. Hence, of the research ideas generated by these psychologists, who are all card-carrying scientists and who liked these ideas well enough to invest weeks or months of their lives working on them, less than 25% of 40% of 10% = 1% actually appear to make some sort of contribution to the discipline.

Given that this work was published in the 1970s when the "publish or perish" pressure was less strongly developed I expect today's number to be worse. Concerning Information Retrieval the field of research of my PhD I would certainly attest that this applies.

Peer review is slow because publishing and retracting used to take a long time, now it is instant. And as for the quality of the reviewer, perhaps it would make sense for the journal to employ professional scientists that only review -- they'll be up to date on all the current research (that's their job), can highlight gaps, and can help cross pollinate other disciplines by reviewing 2 or 3 different but related topics.

The idea that a scientist should do everything seems to be more and more inefficient. We need to break down tasks like we do in the commercial world so that experts can move faster.

You're missing the point of scientific publication. The entire point of publication is so others can reproduce your work. Speed doesn't work if you need a billion data points taken over 20 years to prove a long term issue.

>The idea that a scientist should do everything seems to be more and more inefficient. We need to break down tasks like we do in the commercial world so that experts can move faster.

The problem here is that for a lot of cutting edge theoretical research, the experts are the only ones qualified to really vet the paper. And finding experts that: 1. Are capable of understanding everything in the paper and 2. Are willing to put aside their personal research to review the paper is a VERY small set of people.

For example, the P=NP proofs produced over the last few years were only presented to a small group of about 20 very qualified mathematicians to vet. I, with an undergrad CS degree, could barely understand the summary of the proofs. As far as I know, those proofs still have not been completely refuted or accepted. I don't think that this is inefficient, it's simply the fact that there aren't enough people with enough time to really work with extraordinarily complex concepts and ideas.

Science is also not not-for-profit. Plenty of for-profit businesses do research, and publish in scientific journals. Some research fields (drug development comes first to mind) have a very large industry presence.

http://arxiv.org/pdf/1107.2875.pdf

The scheme you are advocating might have succeeded 200 years ago, when science was happening at a much for fundamental level than it is now. These days, things are so specialized that what the OP says is spot-on: only a handful of people are qualified to knowledgeably comment on your work. What's more, these people are all /incredibly/ busy; I would know since I spend my time trying to get their attention (grad student) :-) In short, the way we conduct research is kind of like what Churchill said about democracy; it's the worst system except for all the other systems.