Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn't preprint review being adopted?

theroadgoeson.com

Post reply on HN
Print view View original

81–90 of 169 posts

Re: Why isn't preprint review being adopted?

#81

Post by bumby »

heikkilevanto wrote: post #62
bumby wrote: post #14

Earlier quoted context omitted.

I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarit…

So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...

Or possibly just a way to review the reviewers. This opens itself up to competitor bias, though, so it would need to be thought out in a way to minimize that.

Re: Why isn't preprint review being adopted?

#82

Post by bumby »

aashiq wrote: post #31
nitwit005 wrote: post #22

Earlier quoted context omitted.

Unfortunately, what you'll actually incentivize is spending as little effort as possible to get the money paid out.

It depends! There should probably also be a process by which reviewers themselves get graded. Then paper writers can choose whether to splurge for fewer really amazing reviewers, or a larger quantity of mediocre reviewers. Also, readers will be able to see the quality of the reviewers that looked at a preprint.

Splurging for "amazing reviewers" could also be gamed to "splurge on those reviewers who are likely to rubber-stamp my submission to get paid" (not unlike some of the questionable open-access journals' current business models).

Re: Why isn't preprint review being adopted?

#83

Post by bumby »

BlueTemplar wrote: post #50
aashiq wrote: post #31

Earlier quoted context omitted.

It depends! There should probably also be a process by which reviewers themselves get graded. Then paper writers can choose whether to splurge for fewer really amazing reviewers, or a larger quantity of mediocre reviewers. Also, readers will be able to see the quality of the reviewers that looked at a preprint.

How do you have all three of anonymous authors, anonymous reviewers, and reviewer ratings ?

It might be possible to have a third-party manage the reviewer ratings. Although I suspect some fields are so small/niche that if someone wanted to associate some random ID with a real person, they could match writing styles etc.

Re: Why isn't preprint review being adopted?

#84

Post by bumby »

rhelz wrote: post #63
geysersam wrote: post #40

Earlier quoted context omitted.

Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential. If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good,…

Whether or not your papers pass peer review—-and which journals it is published in—are important criteria for hiring, tenure, whether your grants are funded, etc. If you get rid of peer review, it’s not science. It’s just a vanity press.

>If you get rid of peer review, it’s not science. It’s just a vanity press.

If you believe replicability is central to science, the current paradigm doesn't necessarily converge on science either. And when people are graded on how many publications they garner, it borders on turning publication into a symbol of status rather than one of science.

Re: Why isn't preprint review being adopted?

#85

Post by bumby »

matwood wrote: post #25
llm_trw wrote: post #3

Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.

You actually touch on an interesting point. Is peer review still necessary? When there is limited journal space, sure. But now that we have effectively unlimited space, let it be reviewed when people go to cite it. We've already seen a lot of peer reviewed papers be retracted later, so should we just accept the reality?

Necessary to share ideas? No.

Necessary to share important ideas? Maybe.

One of the aspects of journals that I don't see talked about much here is the curation of articles. Saying (good) journals aren't necessary is like saying journalists aren't necessary to get your news in the internet era. Not all information is necessarily good, and we often rely on systems/people to help winnow the amount we have to sift through. This is important in a society where the sheer amount of information developed far outpaces what we can consume in several lifetimes.

Re: Why isn't preprint review being adopted?

#86

Post by bumby »

warkdarrior wrote: post #79
bumby wrote: post #21

Earlier quoted context omitted.

I wonder if there's a way to document such replaceability failures as an erratum to the original manuscript. I feel like this would help in at least two major ways: 1) It provides a reproducibility filter. If a method isn't shown to be reproducible, publically documenting that adds to the body of knowledge, and this would help drive an incentive towards reproducing work rather than just searching for novelty. It woul…

2) ???

If you are willing to provide a more substantive response, I might be able to clarify.

Re: Why isn't preprint review being adopted?

#87

Post by cqqxo4zV46cp »

SkyPuncher wrote: post #57
adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

I've found the entire peer review concept in academia to be extremely odd. I'm not entirely sure what problem it solves. It seems like you have one of two types of people reading these articles: * People who are already specialists/familiar enough with concepts. They'll either call BS upfront or run experimentation themselves to validate results. * People who aren't specialists and will need to corroborate evidence a…

Most of the world isn’t ‘about’ software engineering. Software has some attributes that not very many other fields do. And software people have a habit of thinking that their experiencing writing software grants them some sort of transferable expertise, which it certainly does not.

Re: Why isn't preprint review being adopted?

#88

Post by freedomben »

zer00eyz wrote: post #77
freedomben wrote: post #48

Earlier quoted context omitted.

> You are not longer allowed to site a non reproduced paper in yours. I fully agree that in an ideal world, that would be the case. But some reproductions (especially now with machine learning) could cost millions of dollars and years to do. I don't think that's a reasonable or feasible thing to require.

>> could cost millions of dollars and years to do https://en.wikipedia.org/wiki/List_of_colleges_and_universit... Not only do they have the money they are charging the students more than ever.

This is a very fair criticism and something that absolutely should be discussed, although I think it's a separate issue as papers can be written and published by anybody, regardless of backing from academic institutions.

Re: Why isn't preprint review being adopted?

#89

Post by llm_trw »

bumby wrote: post #85
matwood wrote: post #25

Earlier quoted context omitted.

You actually touch on an interesting point. Is peer review still necessary? When there is limited journal space, sure. But now that we have effectively unlimited space, let it be reviewed when people go to cite it. We've already seen a lot of peer reviewed papers be retracted later, so should we just accept the reality?

Necessary to share ideas? No. Necessary to share important ideas? Maybe. One of the aspects of journals that I don't see talked about much here is the curation of articles. Saying (good) journals aren't necessary is like saying journalists aren't necessary to get your news in the internet era. Not all information is necessarily good, and we often rely on systems/people to help winnow the amount we have to sift throug…

It's more like saying that maybe we don't need the NYT, the people who lied about Iraq: https://www.theguardian.com/world/2004/may/27/media.iraq, to get good quality journalism when the internet exists.

Journals are in the same boat. Any good they have done is dwarfed by the few 'honest mistakes', like Alzheimer's cabal: https://www.statnews.com/2019/06/25/alzheimers-cabal-thwarte..., which have done more to stifle science than anyone since Stalin picking which biological theories are socialist enough to be true.

Re: Why isn't preprint review being adopted?

#90

Post by llm_trw »

freedomben wrote: post #41
matwood wrote: post #25

Earlier quoted context omitted.

You actually touch on an interesting point. Is peer review still necessary? When there is limited journal space, sure. But now that we have effectively unlimited space, let it be reviewed when people go to cite it. We've already seen a lot of peer reviewed papers be retracted later, so should we just accept the reality?

Peer review is an important signal to potential citers. If everyone has to fully read and understand every paper in order to (responsibly) cite it, there's gonna be a lot less research. Given the exposure of how much bad research there is, maybe there does need to be a slowing and focus on quality, but I think we still need peer review, although it definitely need to be reformed somehow because it's clearly broken. W…

Peer review can be very simply replaced by skimming the abstract. If the abstract doesn't make sense the paper doesn't make sense either.

All the disciplines which use arxiv as their main journal are doing well enough without peer review.

Post reply on HN