I heard a story the other day about a paper trying to make it into CHI, a big conference in human-computer interaction. In CS and its subfields, conferences like CHI tend to be more important than journals (or so I've heard). Anyway, the paper was rejected by reviewers, and the rebuttal wasn't enough to make it into the conference.
The paper was stellar, and the author knew it, so he persisted, reasoning that the paper was just too far-reaching for his reviewers. He applied to a portion of the conference called alt.chi, where highly externally reviewed papers can be accepted into the conference, despite being turned down by the review proper. He was able to garner 38 positive named reviews - a shoo-in - and was entered into the conference.
On one hand, this is a great story about how the academic community can build processes that correct for conservative or short-sighted peer review. OTO, though, it's interesting to note that if he hadn't had 38 positive named reviews (or some n less than 38, but considered very strong), he wouldn't have been accepted. This is because, historically, named review is messed up- no one would negatively review his paper and attach his or her name. I can't talk too much about the details of arXiv, but I do know that in a named review system, it's easy to both
a) discourage negative commentary, and
b) discourage younger or less experienced reviewers, who are especially afraid of hurting their reputation against more experienced colleagues.
Don't get me wrong- I do think that more open systems can replace the current peer review- but I also believe that double-blind reviews and sometimes the intervention of an editor are important for a coercion-free process. They also make it easier to cut the information clutter- instead of following individual researchers, I can follow a few peer-reviewed outlets I trust.
EDIT: I guess a big part of my point is that while it doesn't need to be traditional peer review, some of the aspects of traditional peer review process need to be maintained in the community.