Why isn't preprint review being adopted?

> If you (somehow) prevented me from distributing my preliminary results

Dude, nobody can prevent you from putting your paper on arxiv. But arxiv is a vanity press-I'm sorry to put it in such negative terms, but that's the God's honest truth of it.

We have free speech; you can get a vanity press to print whatever you want, as fast as you want it to. But that is not doing science. Alas, arxiv is needed, because the rest of the system is so broken. But imagine if you could submit your paper to any journal, and be guaranteed that it would be peer reviewed within 2 weeks. We could do that if we didn't have to rely on volunteer labor, but paid qualified people to do the job thoroughly and on time.

> Having fast and efficient channels for publishing our results

Have you ever submitted a paper to a journal? If so, I'm sure you were as frustrated as I was that my paper was just sitting on reviewers desks for a year before they got around to giving it a cursory glance.

If we actually payed reviewers, we could specify that the reviews must be done on a certain time schedule. My proposal would greatly accelerate the rate at which non-fraudulent, scientific results get published and communicated to other researchers.

> I strongly object to any proposal that would interrupt this essential process.

Its not an essential process to science to pick the fastest and cheapest vanity press.

What essential for science is getting repeatable results. That's science.

> Having fast and efficient channels for publishing

Ever faster publishing of ever more vanity projects is not science, nor does it help science. Quite the opposite.

> it's obvious that when you say publication you don't mean it in the literal sense of communicating scientific results.

No, that's exactly what I mean. But putting something up on arxiv isn't "communicating scientific results." Until it has been peer reviewed and shown to be replicable, it just isn't a scientific result.

> [change metric, etc]

I don't want to change the metric. We don't have a bad metric--we have frauds claiming they have met the metric when they haven't. The problem isn't with the metric, its that the metric isn't actually being enforced like it should be.

> any measure that becomes a target ceases to be a good measure.

Peer reviews and replications are not a measure of how good your science is. The measure for good vs poor science are things like how useful your results are, how general they are, whether they unify previously disconnected areas of knowledge...things like that.

Whether you are generating repeatable results or not isn't the difference between poor science and good science. It's the the difference between doing science and not doing science.

You can't get rid of peer review, and the demand for repeatable results and still be doing science. Science is the the process of getting peer reviewed and repeatable results.

Peer review and repeatability isn't how you judge the results of the race--they are the bare minimum requirements to enter the race to begin with.

> your proposal really has nothing to do with scientific publication for its first-order effects (i.e., distributing results to colleagues)

Distributing what results? If your "results" can't pass peer review, or they can't be replicated, the are not scientific results. If you skip those steps, you are are skipping doing science. You are not a scientist, you are just cosplaying a scientist.

In order to deliver scientific results, quickly or slowly, you actually have to have produced scientific results.

> if you want to reform the way credit is allocated, why not focus explicitly on that goal?

Well, I'm not trying to reform the way credit is allocated. I'm trying to stop people from getting credit for fraudulent results. Before credit is allocated, something credible must have been produced!! And until your paper passes peer review, it is not credible, and until it has been replicated, its not science.

> that real scientists will instantly work around.

Even calling them "real scientists" betrays a deep conceptual error. There are not "real scientists" and "other kinds of scientists." There are scientists, and there are non-scientists.

The distinction isn't between "real scientists" who take every shortcut and cheat as much as they can get away with, and "poor schlubby scientists" who don't have the guts to cheat.

Scientists (not "real scientists", just scientists) insist on peer review and reproducible results. You can't "work around" the most basic criteria for something be scientific and still be a scientist doing science.

> just .... convince them to stop hiring/funding/firing researchers using the metrics they currently use

So...I'm supposed go to the national science foundation, and every body which funds scientific research, every institution which purports to hire scientists---and somehow convince them to stop actually doing science?

sigh its not your fault man. The problem has been going on for so long now, that there are generations of cosplayer-professors, who have been graduating cosplayer-ph.d.s. Imagine people going to Star Trek conventions, dressing up, geeking out---but after few generations they forget that they are cosplaying, and think they are actually on a starship....

Seems ludicrous, but that's kind of what arxiv has inadvertently done. It doesn't help that have people who cosplay being "science" journalists or "science" popularizers, who trawl arxiv for juicy headlines, and happily write up an article about "research" which hasn't been peer reviewed or replicated. It just encourages more race-to-the bottom, by encouraging "researchers" to post exaggerated claims.

I think the idea of requiring the review process to require replication would potentially be a good approach, given we're aware of the downsides. For example, I've worked in labs with sensitive data, or with proprietary processes that they would not want to share. This would mean the advocated process would result in a lot less sharing of methods. Maybe there's a chance there could be vetted independent labs that meet stringent security requirements, but that adds another layer of bureaucracy which could, again, result in less sharing of information. There's a balancing act to be considered, and I agree that we are probably too far on the one side of that equation currently.

Most of your rebuttals seem to hinge on "yeah, but that problem isn't unique to publishing." That is a kind of side-stepping that misses the point. The point is we need to create a system that mitigates those downsides, not ignore them. I I don't think a store manager would be okay saying, "Well, people steal from all kinds of stores, so we don't need to try to minimize theft." They recognize stealing is a natural outcome given human tendencies and create a system to minimize it within reasonable boundaries.

> Most of your rebuttals seem to hinge on "yeah, but that problem isn't unique to publishing." That is a kind of side-stepping that misses the point.

If there is a specific objection you'd like to revisit, I'd be happy to discuss it. But I wouldn't self-describe what I'm doing as "sidestepping"--I'd say its avoiding bike shedding and keeping the conversation focused.

I mean, it's a pretty facile objection to say some variation on "but if we pay them how do we know we'll get our money's worth?" when we pay for goods and services all the time with very high confidence that we'll get what we pay for.

Surely, there's plenty of considerations to discuss, and I've tried to squarely address all objections which are specific to this proposal. But how to hire and use consultants, or how to ensure you get what you contracted for, are largely solved problems, and off-topic.

> This would mean the advocated process would result in a lot less sharing of methods.

I don't think my proposal would even apply to internal R&D groups who wanted to keep things proprietary. I mean, I can certainly understand wanting to reserve some methods or data as being proprietary. But choosing to do that is, ipso facto, not sharing them. How would paying reviewers and replicators for their time cause any less sharing to happen?

I mean, if your paper doesn't describe the experiments you performed in enough detail to allow other groups to replicate it, its not a scientific paper to begin with. It's either a press release, or a whitepaper, or some other form of creative writing, and publishing it is either public relations, or advertisement--not science.

Which is not to say that it's immoral or useless, or to denigrate it in any way. Not everything we do has to be science. My proposal is just for scientists communicating scientific results with other scientists. Maybe I'm missing something, but I don't see how it would inhibit the kinds of practices you are describing in any way.

It would make it harder for people to claim their "results" are scientific, but are not. It would be a big obstacle to publishing fraudulent papers in scientific journals. It would make it harder for somebody to claim the mantel of "science" to give credibility to their claims. But I really don't see how paying reviewers and replicators would stop anybody from sharing as much or as little as they wanted to.

How isn't that third party just reinventing journals ?