Let’s make peer review scientific

As an academic in biomedical sciences and as someone who tried to create a journal that did all the things the OP said, thank you.

The old crappy system is mandatory. You can't decide to do otherwise and still have a job.

The impetus for change needs to start at the funding source level, NIH. They need to make a journal that one is required to publish in order to keep your funding.

In mathematics, Perelman posted his beautiful 3 part proof to Poincare to arXiv and within a day there was allegedly hype (or so reputed professors report, while I did go into advanced alg. top. later, I was 14 at the time). Professors (the same professors who'd referee) were independently reviewing Part 1 within 24 hours and this was in 2002.

Then you have a Fields Medal winner Voevodsky who also bitches about having a paper stuck in peer-review limbo for three years.[0]

In cosmology (I am not a physicist in the slightest, but I have enough mathematical background s.t. I can watch lower graduate level classes without too much difficulty in a recreational capacity), pre-prints get e-mailed around daily, such that if Lenny Susskind posts something on evening-{n}, morning-{n+1} will have most of the field commenting on it.

In biological sciences (again, just from hear say mind you) I understand it's different (e.g. in medicine, specifically, my uncles/father/their colleagues and students are all concerned with the impact factor of their papers and/or their doctoral students papers).

I also realize some journals have a "no preprint" policy. But a lot of heavy hitters (Nature, Elsevier, Springer[1]) allow the circulation of pre-prints now. So it's not a binary decision between one or the other in some cases. (And in the instances where you are able to submit pre-prints to online, if you've made a mistake, a colleague could inform you and you can issue a retraction hopefully before the $month+1's Nature Methods hits physical press.)

---

Side-bar, skip over if you don't care about proof theory, type theory, constructive logic, univalent foundations, or category theory

[0] Vlad on Univalent Foundations http://www.math.ias.edu/vladimir/files/univalent_foundations...

Interestingly enough, for you category theory fans, Vlad has recently become a huge proponent of Homotopy theory[0.b]. Everyone knows Wildberger has kookie ideas w/r/t infinitesimals but Vlad too shares the same 'feeling[0.a]' that analysis (and to a lesser extent axiomatic set theory) just doesn't 'feel' foundationally correct. He's a huge proponent of Coq, Agda and dependent typing instead. Princeton's IAS put out the seminal text that's worth a read. No advanced mathematics required (or even advanced CS knowledge, it's all self-contained).

[0.a] I can't really put into words what the 'feeling' is any more than describing why some people find algebraic proofs more elegant than analytic or geometric. But in the same way that there's mathematical beauty [0.a.i] well agreed upon, this 'gut feeling' too is agreed upon amongst those who fall into a certain category (pun not intended)

[0.a.i] https://en.wikipedia.org/wiki/Mathematical_beauty [0.b] https://homotopytypetheory.org/book/ -- think of this as the best minds in aggregate sit down at Princeton to write one of the best texts on Proof theory, type theory and logic. Practical Foundations for Programming Languages (Robert Harper) is a good companion text.

[1] https://en.wikipedia.org/wiki/List_of_academic_journals_by_p...

arxiv is not open. At least not as open as it should be in my opinion. But I fully agree on the chicken-or-egg problem. The main problem is bootsrapping such web service.

I don't think that "anoiting" a forum/web service would be a good strategy. We need to keep open the possibility of a new and better system to take over.

My feeling is that two fundamental requirements must be satisfied. 1. Anybody should be able to submit a paper. 2. Anybody must have an opportunity to become a recognized expert in a field. By anybody I mean that academic or career credentials must not come into play.

Obviously this begs for a rating system on the articles and on the "experts" too. Obviously recognized "experts" should have a bigger weight in the rating than random members. This makes it different from reddit and hacker news. But the idea is similar. Members should also be able to filter papers by rating and be notified only when a paper reaches a desired ranking. This is another difference from hacker news and reddit.

I would be very interested in participating in a project of this kind.

Science (Doing and reviewing it)is an art, not a science.

It could work more like the open source software project model: you would get your respect from having several "accepted pull requests" to the literature. The difference is that your PRs would be better framed in terms of identifying your specific contribution.

As it stands now, the publishing model is the equivalent of writing your own version of the project -- with no easy metadata about your specific innovation -- and asking it to be accepted or rejected as a whole.

With that said, it's understandable why to keep doing the latter version until academia has standardized on the former.