Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why isn't preprint review being adopted?

theroadgoeson.com

Post reply on HN
Print view View original

11–20 of 169 posts

Re: Why isn't preprint review being adopted?

#11

Post by Muller20 »

s0rce wrote: post #7
llm_trw wrote: post #3

Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.

While not always true my metric for clear/understandable is for other people to understand it. This usually supports my argument when people show me a document and I have no idea what its saying, they argue its perfectly clear... my definition was for people other than the author to grasp the intended meaning.

Peer review goes beyond simple issues about clarity or misunderstanding. In particular, peer review is sometimes seen as an adversarial process.

Often, the reviewer will not understand because he is not the intended audience. Other times, he will understand but he just doesn't like your method, because he is working in an opposite direction. Or maybe your method is a direct competitor of his and yours work better, which incentivizes some people to block your work.

Re: Why isn't preprint review being adopted?

#13

Post by wrycoder »

Muller20 wrote: post #11
s0rce wrote: post #7

Earlier quoted context omitted.

While not always true my metric for clear/understandable is for other people to understand it. This usually supports my argument when people show me a document and I have no idea what its saying, they argue its perfectly clear... my definition was for people other than the author to grasp the intended meaning.

Peer review goes beyond simple issues about clarity or misunderstanding. In particular, peer review is sometimes seen as an adversarial process. Often, the reviewer will not understand because he is not the intended audience. Other times, he will understand but he just doesn't like your method, because he is working in an opposite direction. Or maybe your method is a direct competitor of his and yours work better, wh…

Or your paper goes against the current paradigm or is otherwise politically unpalatable.

Re: Why isn't preprint review being adopted?

#14

Post by bumby »

rhelz wrote: post #10

One of my profs once remarked, "All of science is done on a volunteer basis." He was talking about peer review, which--as crucial as it is--is not something you get paid for. Writing a review--a good review--is 1) hard work, 2) can only be done by somebody who has spent years in postgraduate study, and 3) takes up a lot of time, which has many other demands on it. The solution? Its obvious. In a free market, how do y…

I don't disagree with the proposed idea of paying for review, but I would prefer also to have guardrails to ensure a good review. I would be willing to pay for a good review because it makes the paper/investigation better. But let's face it: under the current paradigm, there are also a lot of really bad reviews. It's one thing when it's apparent that a reviewer doesn't understand something because of a lack of clarity in the writing. But it's also extremely frustrating when it's obvious the reviewer hasn't even bothered to carefully read the manuscript.

Under a payment paradigm, we need mechanisms to limit the incentive to maximize throughput as a means of getting the most pay and instead maximize the review quality. I assume there'd be good ways to do that, but I don't know what those would be.

Re: Why isn't preprint review being adopted?

#15

Post by bumby »

zer00eyz wrote: post #5

The review process is broken. Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar. Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from. The current system is glad-handing, intellectual protectionism and mastrubation. A…

I think we need to find ways of giving status to reproducing studies. Maybe not as much as novelty, but definitely something greater than what it is currently.

Re: Why isn't preprint review being adopted?

#16

Post by bumby »

s0rce wrote: post #7
llm_trw wrote: post #3

Because I don't want to spend two years fighting the second referee when she's fundamentally misunderstood the point of my paper. I'm no longer in academia. Either take what I put up on arxiv or leave it. I _really_ don't care.

While not always true my metric for clear/understandable is for other people to understand it. This usually supports my argument when people show me a document and I have no idea what its saying, they argue its perfectly clear... my definition was for people other than the author to grasp the intended meaning.

If the intended audience is reading it, I would agree. However, many reviewers seem to be assigned and agree to review a topic that they are ill-equipped to understand without a substantial amount of background knowledge.

A good reviewer in this situation will review the referenced papers to bolster their understanding. A bad reviewer will expect everything to be spelled out within the manuscript, and, unfortunately, the length limits often don't permit that kind of write-up.

Re: Why isn't preprint review being adopted?

#17

Post by [deleted] »

wrycoder wrote: post #13
Muller20 wrote: post #11

Earlier quoted context omitted.

Peer review goes beyond simple issues about clarity or misunderstanding. In particular, peer review is sometimes seen as an adversarial process. Often, the reviewer will not understand because he is not the intended audience. Other times, he will understand but he just doesn't like your method, because he is working in an opposite direction. Or maybe your method is a direct competitor of his and yours work better, wh…

Or your paper goes against the current paradigm or is otherwise politically unpalatable.

[deleted]

Re: Why isn't preprint review being adopted?

#18

Post by glial »

bumby wrote: post #15
zer00eyz wrote: post #5

The review process is broken. Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar. Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from. The current system is glad-handing, intellectual protectionism and mastrubation. A…

I think we need to find ways of giving status to reproducing studies. Maybe not as much as novelty, but definitely something greater than what it is currently.

IMO reproducing findings could/should be a mandatory part of PhD training.

Re: Why isn't preprint review being adopted?

#19

Post by barfbagginus »

timr wrote: post #2

Even if academics could review all the papers on a preprint server (which the article argues -- rightly -- that they can't), it wouldn't solve the perceived problems (or the actual problems) with scientific peer review. The vast majority of irreproducible papers aren't detectible as irreproducible at time of publication. They look fine, and many actually are fine . They just don't reproduce. That's an expected outcom…

In AI work - which naturally lends itself to replicability improvements - we could get truly solid replicability by ratcheting up the standards for code quality, testing, and automation in AI projects. And I think llms can start doing a lot of that kind of QA and engineering / pre-operationalization work, because the best llms are already better at software and value engineering than the average postdoc AI researcher.

Most AI codes are missing key replicability factors - either the training data/trainer are missing, the code has a poor testing / benchmark automation strategy, the code documentation is meager, or there's no real CI/CD practice for advancing the project or operationalizing it against problems caused by the anthropocentric collapse.

Some researchers are even hardened against such things, seeing them as false worship of harmful business metrics, rather than a fundamental duty that could really improve the impact of their research, and it's applicability towards a universal crisis that faces us all.

But we can put the lie to this view with just one glance at their code. Too much additional work is necessary to turn it into anything useful, either for further research iterations or productive operationalization. The gaps in code quality exist not because that form of code is optimal for research aims, but because researchers lack software engineering expertise, and cannot afford software engineering labor.

But thankfully the level of software engineering labor is not even that great - llms can now help swing that effort.

As a result I believe that we should work to create standards for AI assisted research repos that correct the major deficits of replicability, usability, and code quality that we see in most AI repos. Then we should campaign to adopt those standards into peer review. Let one of the reviewers be an AI that really grills your code on its quality. And actually incorporate the PRs that it proposes.

I think that would change the situation, from the current standard where academic AI repos are mainly nonreplicating throw-away code, to an opposite situation where the majority of AI research repos are easy to replicate, improve, and mobilize against the social and environmental problems facing humanity, as it navigates through the anthropocene disaster.

Re: Why isn't preprint review being adopted?

#20

Post by ajmurmann »

zer00eyz wrote: post #5

The review process is broken. Reviewing pre print papers isnt any more effective than reviewing printed papers. Review, and publication is a meaningless bar. Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from. The current system is glad-handing, intellectual protectionism and mastrubation. A…

"Publish -> people find insight and try to pick it apart -> You either have flaws or you get reproduced... Only then should your paper be of any worth to be quoted or sighted from."

This is already how it's supposed to work. The review before publication is a fairly superficial check that just confirms that what you describe follows basic scientific practices. There is no validation of the actual research. A proper reproduction is what's supposed to come after publication.

IMO the real problems are that a) there isn't much glamour and funding for reproducing other's studies and b) "science journalists", university PR departments and now in part people on social media are picking up research before people on the field looked at it or misrepresent it. Suddenly the audience is a lot of folks who never were the intended audience of the process.

Post reply on HN