Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn't preprint review being adopted?

theroadgoeson.com

Post reply on HN
Print view View original

31–40 of 169 posts

Re: Why isn't preprint review being adopted?

#31

Post by aashiq »

nitwit005 wrote: post #22
rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

It depends! There should probably also be a process by which reviewers themselves get graded. Then paper writers can choose whether to splurge for fewer really amazing reviewers, or a larger quantity of mediocre reviewers. Also, readers will be able to see the quality of the reviewers that looked at a preprint.

Re: Why isn't preprint review being adopted?

#32

Post by dbingham »

_delirium wrote: post #27

Does openreview.net not count as preprint review in the sense the author means? It has substantial uptake in computer science.

Nice catch! I was going from the data shared in that paper[1] and didn't notice that it excluded OpenReview.net (which I'm aware of). The paper got their data[2, 3] from Sciety and it looks like OpenReview isn't included in Sciety's data.

It may have been excluded because OpenReview (as I understand it) seems to be primarily used to provide open review of conference proceedings, which I suspect the article puts in a different category than generally shared preprints.

But it would be worth analyzing OpenReview's uptake separately and thinking about what it's doing differently!

[1] https://journals.plos.org/plosbiology/article?id=10.1371/jou...

[2]https://zenodo.org/records/10070536

[3] https://lookerstudio.google.com/u/0/reporting/b09cf3e8-88c7-...

Re: Why isn't preprint review being adopted?

#33

Post by michaelt »

andyferris wrote: post #29
matwood wrote: post #24

Earlier quoted context omitted.

Opposed to now where it appears lots of science is peer reviewed with all the problems found later?

It might cost $100k - $1m (or more) to repeat the work and run replications. The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper.

> The $5k - $10k mentioned earlier would be enough to allow time reading and thinking and checking some stuff on pen-and-paper.

The average postdoc in the US earns $32.81/hour, according to the results of some googling. Even taking overheads into account, $5k should cover more than a week's full time work.

Re: Why isn't preprint review being adopted?

#34

Post by eviks »

rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

The outcome would most likely be exactly this: "it's probably research not worth doing in first place" (and why would you want to signal you want more busy work?)

Re: Why isn't preprint review being adopted?

#35

Post by dazed_confused »

matwood wrote: post #25
llm_trw wrote: post #3

Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.

You actually touch on an interesting point. Is peer review still necessary? When there is limited journal space, sure. But now that we have effectively unlimited space, let it be reviewed when people go to cite it. We've already seen a lot of peer reviewed papers be retracted later, so should we just accept the reality?

Yes. The crap people publish even with peer-reviews is bad enough. A few subfields in CS have bitten a couple times by relying on non peer reviewed ecosystems.

Re: Why isn't preprint review being adopted?

#36

Post by [deleted] »

wrycoder wrote: post #13
Muller20 wrote: post #11

Earlier quoted context omitted.

Peer review goes beyond simple issues about clarity or misunderstanding. In particular, peer review is sometimes seen as an adversarial process. Often, the reviewer will not understand because he is not the intended audience. Other times, he will understand but he just doesn't like your method, because he is working in an opposite direction. Or maybe your method is a direct competitor of his and yours work better, wh…

Or your paper goes against the current paradigm or is otherwise politically unpalatable.

[deleted]

Re: Why isn't preprint review being adopted?

#37

Post by hiddencost »

nitwit005 wrote: post #22
rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

I hear this sentiment a lot.

There was a time when academia was intensely driven by culture. People did stuff because they cared about the community surviving.

It is, in fact, possible to choose the "cooperate" triangle of the prisoner's dilemma, but it takes a lot of work, and is vulnerable to being wrecked by people who don't care / believe in the goals.

Re: Why isn't preprint review being adopted?

#38

Post by adastra22 »

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT.

Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published papers by sending it off as a letter to the editor (sometimes via a sponsor if you were new to the journal), and they either accepted or rejected it as-is.

The idea of having your intellectual competitors review your work and potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective.

The rise of Arxiv is a recognition by researchers that we don’t need or want that system, and we should do away with it entirely in this era of digital print where page counts don’t matter. So please stop trying to force it back on us!

Re: Why isn't preprint review being adopted?

#39

Post by gwerbret »

matwood wrote: post #25
llm_trw wrote: post #3

Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.

You actually touch on an interesting point. Is peer review still necessary? When there is limited journal space, sure. But now that we have effectively unlimited space, let it be reviewed when people go to cite it. We've already seen a lot of peer reviewed papers be retracted later, so should we just accept the reality?

Peer review is the backbone of journals, and (some) journals serve an absolutely critical function in academia: they increase the signal:noise ratio. Most of what is published is noise; without the curation provided by (certain) journals, anything of significance is very likely to be drowned out.

As a casual example in the biomedical sciences, the Journal of Biological Chemistry has an output of ~30,000 pages per year, most of which is 'noise'. That's just ONE journal. The journal Cell, on the other hand, has an order of magnitude less, most of which is 'signal'.

EDIT: This is not to say the peer review approach doesn't need work, and lots of it. The whole current approach to research needs an overhaul. I'm just saying it's a bit hasty to throw the baby out with the bath water.

Re: Why isn't preprint review being adopted?

#40

Post by geysersam »

rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential.

If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good, if they don't they can read the old version.

Peer review as a filter for publishing is terrible in a time when making a few megabytes of text and images accessible is literally free. If anyone wants to run a content aggregator (aka a Journal) they just do it. If they want to change something about the article before it's approved for the aggregator they can contact the authors or ask someone to review it or whatever.

Just make it accessible.

Post reply on HN