Harnessing the Universal Geometry of Embeddings

This is how the academic dialogue works, and it dates back to early durable records in the ancient world -- about as far before Plato as Plato's Academy is before today's universities. Innovations like academic journals come from "merely" four centuries ago, and pre-publication evaluation by peers about three hundred years ago. Anonymized pre-publication review is more recent still, and publishing schedules (as well as the needs of the submitter to have the submitted work dealt with in good time) don't admit a dialogue over some subtle point a referee takes some issue with. Moreover, such a pre-publication dialogue is generally non-public, and often also filtered through an editor. It can get messy [1].

As I'm sure you've discovered in your own post-secondary education, academia is very conservative (in the sense of "retaining what mostly works, even in the face of mounting scaling problems", foremost among those being the ever-increasing volume of publishable works generated by researchers).

Your own idea is quite conservative, and has been tried from time to time. Direct compensation by journals of referees for their time has been the subject of rigorous human experimentation: https://10.1097/CCM.0000000000006637 https://www.biorxiv.org/content/10.1101/2025.03.18.644032v1 as recent examples, with mixed results.

Direct compensation of referees by journals has a number of pitfalls ranging from the fiddly details of how to deal with income tax liability pay (and credits towards future publication fees, etc) incurs to whether it compromises the objectivity of pre-publication evaluation (which is frankly already not very good in highly specialized areas).

Payment of reviewers by paper submitters is imho an even worse idea -- how do you preserve the anonymity of referees (especially with respect to the submitters)? Do submitters or reviewers have exposure to their (probably different) tax authorities for these transactions? (I can think of at least four national tax agencies that would view the received pay as taxable income, and also potentially subject to value-added tax on professional services. One already sees this problem with conference honoraria.)

Another -- one that already exists in the real world -- might be for institutions and grant-funders to build in support for peer review of others' work by project members. To the extent that detailed pre-publication review benefits academia, and societies that fund academia, as a whole, maybe referees should be rewarded (or even obliged) to do paid reviewing duty. One might compare the requirement to do pro bono work imposed on many practising lawyers by their regulating authorities (especially interesting are jurisdictions in which a party losing a court contest to a party represented pro bono usually must pay into a fund that supports the overall scheme).

Personally I think it would be more useful to everyone to have a solid reply paper published in due course than a blocked initial publication. My ideal is that referees aim to help submitters not accidentally professionally embarrass themselves.

Then on your reproduction idée fixe, supporting the generation of papers which confirm reproduction is something that societies could certainly work on. Many incentives to write and submit papers calling out flaws in others' work already exist, and of course could be expanded. I don't think any of that should be done with masking of the participants. Additionally, reproduction of results is often less interesting than (sometimes much) later testing of published results in significantly different ways.

(Indeed, foundational papers from more than ninety years ago are often re-proven by modern-day experiments, both as a side effect of (or opportunity arising from) pursuing the primary research goal, and as a way of testing e.g. new metrology technology, new computational tools, mathematical advances, and so on. Also, some of those foundational papers simply could not be tested with great accuracy with tools existing at the time of publication. Their publication, however, tended to drive the development of such tools, and in modern days such foundational papers are found to be good only within certain limits that could not have been explored closer to the time of publication.)

- --

[1] https://theconversation.com/hate-the-peer-review-process-ein... (2014)

W.R.T. the grand historical tradition, etc, note that the great majority of people who practiced that were, like Plato, independently wealthy and otherwise idle. Those who didn't need any money.

That's why we give Judges, teachers, and Professors tenure, a guaranteed job for life. People are more likely to be fair judges of other people's performance when they don't need the money.

So I guess if you want a non-conservative fix, it would be universal guaranteed minimum income. Something we already have, BTW, but you have to be over 62 to get it.

We've tried to open it up to more than just the 1%, but we didn't really go all the way. It is really unreasonable to expect somebody who isn't independently wealthy to do free work. It is a lot of work to properly review a paper, and reproducing the results is something you simply can't do at all if you are supposed to already be working 70 hours a week generating your own research.

Of course a researcher paying reviewers would be a disastrous idea. But if the grant-awarding body would hire 3rd parties to validate that their money was actually well spent, I think it would be a win/win.

Thanks for the references on where paying reviewers has already been tried. I'd like to see others reproduce their success :-)