Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn't preprint review being adopted?

theroadgoeson.com

Post reply on HN
Print view View original

41–50 of 169 posts

Re: Why isn't preprint review being adopted?

#41

Post by freedomben »

matwood wrote: post #25
llm_trw wrote: post #3

Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.

You actually touch on an interesting point. Is peer review still necessary? When there is limited journal space, sure. But now that we have effectively unlimited space, let it be reviewed when people go to cite it. We've already seen a lot of peer reviewed papers be retracted later, so should we just accept the reality?

Peer review is an important signal to potential citers. If everyone has to fully read and understand every paper in order to (responsibly) cite it, there's gonna be a lot less research. Given the exposure of how much bad research there is, maybe there does need to be a slowing and focus on quality, but I think we still need peer review, although it definitely need to be reformed somehow because it's clearly broken.

We need to take a serious look at the incentive structure in academia because it's not guaranteeing the scientific results that we expected it to. I don't think we should just abandon the system though.

Re: Why isn't preprint review being adopted?

#42

Post by adastra22 »

matwood wrote: post #25
llm_trw wrote: post #3

Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.

You actually touch on an interesting point. Is peer review still necessary? When there is limited journal space, sure. But now that we have effectively unlimited space, let it be reviewed when people go to cite it. We've already seen a lot of peer reviewed papers be retracted later, so should we just accept the reality?

No, it is not necessary. Just as it was not necessary for the first 300 years of the Royal society’s existence. “Peer review” used to be just a stand in phrase for the marketplace of ideas—seeing how your peers evaluate your work in a public process of getting published and sending letters to the editor. Only in living memory has it been warped into this pre-publication gatekeeping process.

Re: Why isn't preprint review being adopted?

#43

Post by freedomben »

adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

> The idea of having your intellectual competitors review your work potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective.

If this is true (and I'm not doubting you, just acknowledging that I'm taking your word for it) then why abandon the entire system? Why not just roll it back to the state before we added the intellectual competitor review?

Re: Why isn't preprint review being adopted?

#44

Post by ccppurcell »

Peer review is pretty unpopular round these parts. In mathematics/TCS I've had mostly good experiences. Actually most of the time the review process improved my papers.

Clearly something is rotten about the way peer review is implemented in the empirical sciences. But think of all those high profile retractions you read about these days. Usually that comes about by a sort of post hoc peer review, not by anything resembling market forces.

Re: Why isn't preprint review being adopted?

#46

Post by frozenport »

freedomben wrote: post #43
adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

> The idea of having your intellectual competitors review your work potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective. If this is true (and I'm not doubting you, just acknowledging that I'm taking your word for it) then why abandon the entire system? Why not just roll it back to the state before we added the…

>> Why not just roll it back to the state before we added the intellectual competitor review?

Journals don't add much value outside of their peer review.

Most researcher don't care about the paper copies, or pagination, or document formatting services provided by publishers. Their old paper based distribution channels are simply not used.

Re: Why isn't preprint review being adopted?

#47

Post by _delirium »

dbingham wrote: post #32
_delirium wrote: post #27

Does openreview.net not count as preprint review in the sense the author means? It has substantial uptake in computer science.

Nice catch! I was going from the data shared in that paper[1] and didn't notice that it excluded OpenReview.net (which I'm aware of). The paper got their data[2, 3] from Sciety and it looks like OpenReview isn't included in Sciety's data. It may have been excluded because OpenReview (as I understand it) seems to be primarily used to provide open review of conference proceedings, which I suspect the article puts in a…

I do agree it's a bit different. How close maybe depends on what motivates you to be interested in the preprint review model in the first place? Could imagine this varies by person.

In a certain sense, the entire field of comp sci has become reorganized around preprint review. The 100% normal workflow now is that you first upload your paper to arXiv, circulate it informally, then whenever you want a formal review, submit to whatever conference or journal you want. The conferences and journals have basically become stamp-of-approval providers rather than really "publishers". If they accept it, you edit the arXiv entry to upload a v2 camera-ready PDF and put the venue's acceptance stamp-of-approval in the comments field.

A few reasons this might not fit the vision of preprint review, all with different solutions:

1. The reviews might not be public.

2. If accepted, it sometimes costs $$ (e.g. NeurIPS has a $800 registration fee, and some OA journals charge APCs).

3. Many of the prestigious review providers mix together two different types of review: review for technical quality and errors, versus review for perceived importance and impact. Some also have quite low acceptance rates (due to either prestige reasons or literal capacity constraints).

TMLR [1] might be the closest to addressing all three points, and has some similarity to eLife, except that unlike eLife it doesn't charge authors. It's essentially an overlay journal on openreview.net preprints (covers #1), is platinum OA (covers #2), and explicitly excludes "subjective significance" as a review criterion (covers #3).

[1] https://jmlr.org/tmlr/

Re: Why isn't preprint review being adopted?

#48

Post by freedomben »

zer00eyz wrote: post #23
ajmurmann wrote: post #20

Earlier quoted context omitted.

"Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from." This is already how it's supposed to work. The review before publication is a fairly superficial check that just confirms that what you describe follows basic scientific practices. There is no validation of the actual research. A proper r…

There is a pretty simple way to change all of that. Academic standards: You are not longer allowed to site a non reproduced paper in yours. Citations matter as much as the print, put the hurdle there and all of a sudden things will change real quick.

> You are not longer allowed to site a non reproduced paper in yours.

I fully agree that in an ideal world, that would be the case. But some reproductions (especially now with machine learning) could cost millions of dollars and years to do. I don't think that's a reasonable or feasible thing to require.

Re: Why isn't preprint review being adopted?

#49

Post by ramblenode »

timr wrote: post #2

Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine . They just don't reproduce. That's an expected outcom…

> The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine. They just don't reproduce. That's an expected outcome in science.

This is not entirely true. A power analysis is how you determine reproducibility, and researchers should be doing it before they begin collecting data. Reviewers can do it post-hoc with assumptions about the expected effect size (which might come from similar studies). False positives produce inflated effect sizes, so if a result is marginally significant but shows a large effect, that is a good heuristic the result will not reproduce.

Re: Why isn't preprint review being adopted?

#50

Post by BlueTemplar »

aashiq wrote: post #31
nitwit005 wrote: post #22

Earlier quoted context omitted.

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

It depends! There should probably also be a process by which reviewers themselves get graded. Then paper writers can choose whether to splurge for fewer really amazing reviewers, or a larger quantity of mediocre reviewers. Also, readers will be able to see the quality of the reviewers that looked at a preprint.

How do you have all three of anonymous authors, anonymous reviewers, and reviewer ratings ?
Post reply on HN