Why isn't preprint review being adopted? • Hacker News Board

Re: Why isn't preprint review being adopted?

#111

Post by geysersam » Mon, Mar 25, 2024, 6:31 AM UTC

ska wrote: post #67

geysersam wrote: post #40

Earlier quoted context omitted.

Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential. If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good,…

> Just make it accessible. We already have that system, it's called the internet. Nothing stops you or I from putting our ideas online for all to read, comment on, update, etc. The role of the publishers, flawed as it is, has little to do with the physical cost of producing or providing an article, and is filling (one can argue badly) a role in curation and archival that is clearly needed. Any proposal to change the…

That's just not true. Most publicly fundes research is hidden behind paywalls.

Re: Why isn't preprint review being adopted?

#112

Post by geysersam » Mon, Mar 25, 2024, 6:36 AM UTC

rhelz wrote: post #63

geysersam wrote: post #40

Earlier quoted context omitted.

Or, research just gets published online free of charge for everyone to access, and important work will prove itself over time by being discovered, discussed and by becoming influential. If anyone wants to change something about an article (the writing, the structure of the paper, or anything else a reviewer might want to edit) they can just do it and publish a new version. If people like the new version better, good,…

Whether or not your papers pass peer review—-and which journals it is published in—are important criteria for hiring, tenure, whether your grants are funded, etc. If you get rid of peer review, it’s not science. It’s just a vanity press.

And the current system is a holy grail that can never change? You can still based your hiring based on reviews made by peers

Re: Why isn't preprint review being adopted?

#113

Post by adastra22 » Mon, Mar 25, 2024, 7:25 AM UTC

dbingham wrote: post #72

adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

Hi! Author here. Preprint review as it is being discussed here is post-publication. The preprint is shared first, and review layered on top of it later. Click through to the paper[1] I'm responding to and give it a read. But, also, prepublication review doesn't need to be "reintroduced". It's still the standard for the vast majority of scholarship. By some estimates there are around 5 million scholarly papers publish…

Thanks for your charitable response to a reader who quite obviously misunderstood what you were writing about. You might want to update your article to make that clear, as "preprint review" isn't a term I had ever encountered before, and a simple reading of your article doesn't obviously indicate that you were talking about post-publication review. (Note that outside of academia, upload to arXiv is publication, so yes your "preprint review" would be a post-publication review.)

Re: Why isn't preprint review being adopted?

#114

Post by adastra22 » Mon, Mar 25, 2024, 7:37 AM UTC

nextos wrote: post #69

adastra22 wrote: post #66

Earlier quoted context omitted.

That’s a better system. But why involve the journal in review at all? Journals should go back to just publishing papers and any unsolicited letter-to-the-editor reviews, reproductions, or commentary they receive in response. Why add a burden of unpaid work reviewing every single paper that comes through?

I believe ELife may eventually get deeper paid reviews. That is a reason to involve the journal in this process. The way reviews at ELife work can be seen as solicited letter-to-the editor reviews, as these do not influence the outcome. Your article is already published.

Maybe I'm not explaining my point very well. I think the fact that some papers don't get reviewed, even post-publication, is a feature not a bug. Otherwise it starts to present an unnecessary and damaging hurdle to publication, even if the reviews are post-publication because it is an implicit promise of future work.

A proper scientific journal is an efficient clearinghouse for information, in the same way that Hacker News is an efficient clearinghouse for tech news and commentary. Journal editors play the role of dang in this setup, applying a minimal but necessary amount of moderation and editorial decisions. But imagine if you weren't allowed to post a comment to HN unless you first lined up 2-3 high-karma individuals to provide a thoughtful reply. No doubt the result would be high quality discussion, but there'd only be a handful of comments on even front-page posts, and I suspect that the overall value of the site would be vastly less than the present HN.

There should be as few as possible editorial hurdles to clear in order to publish in a journal, e.g. one of the co-authors having published in the journal before or being sponsored by a respected person in the field. And the review that occurs before publication should consist of (1) spellchecks and such, and (2) formatting. This is how science worked for hundreds of years, and there's no reason we can't continue operating this way now that journal page lengths are a non-issue.

(I'm not a reactionary though. I'd like to improve upon the old format in many ways. In particular I'd want journals to provide specific support for publishing reproductions of existing work or 3rd party submitted supporting documents, and much better methods for retracting or correcting a paper.)

Re: Why isn't preprint review being adopted?

#115

Post by adastra22 » Mon, Mar 25, 2024, 7:39 AM UTC

abhgh wrote: post #109

kelipso wrote: post #91

Earlier quoted context omitted.

> run experimentation themselves to validate results Let's just breeze through an that like it's nothing haha. Also the people reading these peer reviewed articles range from new grad students to researchers with decades of experience. There are many ways to see whether an article is high quality of not, which includes peer review, the journal it's published in, the research lab that wrote the paper. Reading a paper…

Agree. With more compute-intensive experiments in ML, I am not sure if any reviewer re-runs experiments. The reviewing is already voluntary - who's going to pay out of their pocket for the compute? It's not feasible.

Re-running experiments is out of scope for any reviewer. You're supposed to trust their data, but review that their conclusion is actually supported by the data, as shown.

Re: Why isn't preprint review being adopted?

#116

Post by adastra22 » Mon, Mar 25, 2024, 7:41 AM UTC

coolhand2120 wrote: post #70

adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

https://en.wikipedia.org/wiki/Replication_crisis The reputation of scientific researchers has been greatly harmed by the current system. Please, help find a way to fix it, or at the very least don't hinder people trying to fix it. Thanks to the way we do things now a coin flip is _better_ than peer review. Public trust in science is at an all time low. I really hope you don't think "this is fine".

> Thanks to the way we do things now a coin flip is _better_ than peer review. Public trust in science is at an all time low. I really hope you don't think "this is fine".

I don't think you read my post? I'm advocating we get rid of the "peer review" [sic] system entirely.

The sibling post is right though that this problem is with bad journalism (and bad institutions), not bad science. People think that "peer review" is actually some sort of scientific hurdle that strengthens the paper. It is not, it was never meant tho fill that role, and has been totally morphed by journalists into something it has no business being.

Re: Why isn't preprint review being adopted?

#117

Post by adastra22 » Mon, Mar 25, 2024, 7:51 AM UTC

bumby wrote: post #80

freedomben wrote: post #43

Earlier quoted context omitted.

> The idea of having your intellectual competitors review your work potentially sabotage your publication prospects as a standard process is a relatively recent addition. And one that has not been shown to actually be effective. If this is true (and I'm not doubting you, just acknowledging that I'm taking your word for it) then why abandon the entire system? Why not just roll it back to the state before we added the…

> Why not just roll it back to the state before we added the intellectual competitor review? The only thing I can currently think of is that the pace of research* has grown so much that a small group of editors may be unable to handle the amount of submissions. This could result in a) an inability of the editors to thoroughly vet the submissions, b) difficulty in "good" submissions being found (ie, separating the whe…

You are mistaken in thinking that editors are supposed to vet submissions. That's not their job! Their job is only to weed out crackpot pseudoscience, which is a much easier task generally and also mostly solved by reputation, or papers outside of the journal's scope. And to edit the remaining papers into a constantly formatted journal publication. And that's done easy enough these days with LaTeX and online printing.

It is NOT the job of journal editors to rate or vet sincere submissions they receive that are on topic for their journal. This only started happening about half a century ago, when demand for publication started to significantly outstrip supply of journal pages, back when journals were actually printed on dead trees and had limited number of pages to keep costs down. The idea then was "we're getting 50 submissions but can only print 12, so let's rate them and pick the best ones." So they started the 'peer review' [sic] process to externalize that vetting cost. It largely didn't exist before then. Only now we can accept all 50, because why not? The marginal cost of one more PDF is practically nil.

Re: Why isn't preprint review being adopted?

#118

Post by MichaelZuo » Mon, Mar 25, 2024, 12:25 PM UTC

currymj wrote: post #71

heikkilevanto wrote: post #62

Earlier quoted context omitted.

So, we just need a meta-review to review the reviews. At a cost, of course. And in order to keep that honest, we need a meta-meta-review...

many CS conferences have something literally called a "meta-review" and then there are further senior people who read and oversee the meta-reviews. it stops there though.

Who picks the meta reviewers?

Re: Why isn't preprint review being adopted?

#119

Post by MichaelZuo » Mon, Mar 25, 2024, 12:30 PM UTC

dbingham wrote: post #72

adastra22 wrote: post #38

PREPUBLICATION REVIEW IS BAD! STOP TRYING TO REINTRODUCE IT. Sorry for the all caps. Publishing papers without “peer review” isn’t some radical new concept—it’s how all scientific fields operated prior to ca. 1970. That’s about when the pace of article writing outstripped available pages in journals and this system of pre-publication review was adopted and formalized. For the first 300 years of science you published…

Hi! Author here. Preprint review as it is being discussed here is post-publication. The preprint is shared first, and review layered on top of it later. Click through to the paper[1] I'm responding to and give it a read. But, also, prepublication review doesn't need to be "reintroduced". It's still the standard for the vast majority of scholarship. By some estimates there are around 5 million scholarly papers publish…

Quality control was handled fine enough by editors for literally all output on the planet pre 1970s.

There’s nothing physically stopping that paradigm from returning now that we have the internet, other than the fact that there’s only a few thousand (?) such editors.

If somehow there were fifty thousand such editors, then the whole peer review system would be completely unnecessary.

Of course not enough people want to pay for that many editors, but that doesn’t stop a partial adoption by those willing to do so via some arrangements.

Re: Why isn't preprint review being adopted?

#120

Post by bumby » Mon, Mar 25, 2024, 2:08 PM UTC

adastra22 wrote: post #117

bumby wrote: post #80

Earlier quoted context omitted.

> Why not just roll it back to the state before we added the intellectual competitor review? The only thing I can currently think of is that the pace of research* has grown so much that a small group of editors may be unable to handle the amount of submissions. This could result in a) an inability of the editors to thoroughly vet the submissions, b) difficulty in "good" submissions being found (ie, separating the whe…

You are mistaken in thinking that editors are supposed to vet submissions. That's not their job! Their job is only to weed out crackpot pseudoscience, which is a much easier task generally and also mostly solved by reputation, or papers outside of the journal's scope. And to edit the remaining papers into a constantly formatted journal publication. And that's done easy enough these days with LaTeX and online printing…

>Only now we can accept all 50, because why not?

Because the downside to creating an ever growing haystack is that it becomes increasingly difficult to find a needle. Making it easier to create a deluge of bad research won’t help me find the worthwhile research that would actually help me in my job.

If I had the choice between collecting “all the data” and just collecting “the really good and relevant data” I’m opting for the latter. You are also contradicting yourself by saying it’s not the editors job to vet submissions, yet also say they weed out “crackpot” work. All you’re saying is they lazily/loosely vet submissions. I’m saying the overall system (not just the editors) have a role in providing a reasonably sized haystack (and would also admit the current system is not great at this, but it’s better than a wild-west approach)