Updated practice for review articles and position papers in ArXiv CS category • Hacker News Board

Re: Updated practice for review articles and position papers in ArXiv CS category

#21

Post by JumpCrisscross » Sat, Nov 01, 2025, 3:27 PM UTC

physarum_salad wrote: post #14

The review paper is dead... so this is a good development. Like you can generate these things in a couple of iterations with AI and minor edits. Preprint servers could be dealing with 1000s of review/position papers over short periods of time and then this wastes precious screening work hours. It is a bit different in other fields where interpretations or know-how might be communicated in a review paper format that i…

> you can generate these things in a couple of iterations with AI

The problem is you can’t. Not without careful review of the output. (Certainly not if you’re writing about anything remotely novel and thus useful.)

But not everyone knows that, which turns private ignorance into a public review problem.

Re: Updated practice for review articles and position papers in ArXiv CS category

#22

Post by physarum_salad » Sat, Nov 01, 2025, 3:27 PM UTC

Maken wrote: post #9

j45 wrote: post #4

Have the papers gotten that good or bad?

Actual papers are as good as ever. This is just trying to stop the flood of autogenated slop, if anything because arXiv hosting space is not free.

It is actually great because it shows how well it works as a system. Screening is really important to keep preprint quality high enough to then implement cool ideas like random peer review/automated reviews etc

Re: Updated practice for review articles and position papers in ArXiv CS category

#23

Post by jvanderbot » Sat, Nov 01, 2025, 3:27 PM UTC

stefan_ wrote: post #7

thomascountz wrote: post #2

The HN submission title is incorrect. > Before being considered for submission to arXiv’s CS category, review articles and position papers must now be accepted at a journal or a conference and complete successful peer review. Edit: original title was "arXiv No Longer Accepts Computer Science Position or Review Papers Due to LLMs"

Isn't arXiv where you upload things before they have gone through the entire process? Isn't that the entire value, aside from some publisher cartel busting?

Almost all CS papers can still be uploaded, and all non-CS papers. This is a very conservative step by them.

Re: Updated practice for review articles and position papers in ArXiv CS category

#24

Post by awestroke » Sat, Nov 01, 2025, 3:27 PM UTC

physarum_salad wrote: post #14

The review paper is dead... so this is a good development. Like you can generate these things in a couple of iterations with AI and minor edits. Preprint servers could be dealing with 1000s of review/position papers over short periods of time and then this wastes precious screening work hours. It is a bit different in other fields where interpretations or know-how might be communicated in a review paper format that i…

A good review paper is infinitely better than an llm managing to find a few papers and making a summary. A knowledgeable researcher knows which papers are outdated and can make a trustworthy review paper, an LLM can't easily do that yet

Re: Updated practice for review articles and position papers in ArXiv CS category

#25

Post by gcr » Sat, Nov 01, 2025, 3:28 PM UTC

jvanderbot wrote: post #13

amelius wrote: post #8

Maybe it's time for a reputation system. E.g. every author publishes a public PGP key along with their work. Not sure about the details but this is about CS, so I'm sure they will figure something out.

Their name, orcid, and email isn't enough?

You can’t get an arXiv account without a referral anyway.

Edit: For clarification I’m agreeing with OP

Re: Updated practice for review articles and position papers in ArXiv CS category

#26

Post by SoftTalker » Sat, Nov 01, 2025, 3:28 PM UTC

amelius wrote: post #8

Maybe it's time for a reputation system. E.g. every author publishes a public PGP key along with their work. Not sure about the details but this is about CS, so I'm sure they will figure something out.

People are already putting their names on the LLM slop, why would they hesitate to PGP-sign it?

Re: Updated practice for review articles and position papers in ArXiv CS category

#27

Post by JumpCrisscross » Sat, Nov 01, 2025, 3:29 PM UTC

physarum_salad wrote: post #22

Maken wrote: post #9

Earlier quoted context omitted.

Actual papers are as good as ever. This is just trying to stop the flood of autogenated slop, if anything because arXiv hosting space is not free.

It is actually great because it shows how well it works as a system. Screening is really important to keep preprint quality high enough to then implement cool ideas like random peer review/automated reviews etc

> we are developing a whole new method to do peer review

What’s the new method?

Re: Updated practice for review articles and position papers in ArXiv CS category

#28

Post by bob1029 » Sat, Nov 01, 2025, 3:29 PM UTC

> The advent of large language models have made this type of content relatively easy to churn out on demand, and the majority of the review articles we receive are little more than annotated bibliographies, with no substantial discussion of open research issues.

I have to agree with their justification. Since "Attention Is All You Need" (2017) I have seen maybe four papers with similar impact in the AI/ML space. The signal to noise ratio is really awful. If I had to pick a semi-related paper published since 2020 that I actually found interesting, it would have to be this one: https://arxiv.org/abs/2406.19108 I cannot think of a close second right now.

All of the machine learning papers are pure slop to me now. The last one I looked at had an abstract that was so long it put me to sleep. Many of these papers aren't attempting basic decorum anymore. Mandatory peer review would fix a lot of this. I don't think it is acceptable for the staff at arXiv to have to endure a Sisyphean mountain of LLM shit. They definitely need to push back.

Re: Updated practice for review articles and position papers in ArXiv CS category

#29

Post by jvanderbot » Sat, Nov 01, 2025, 3:29 PM UTC

efavdb wrote: post #17

arendtio wrote: post #11

I wonder why they can't facilitate LLMs in the review process (like fighting fire with fire). Are even the best models not capable enough, or are the costs too high?

Curious for the state on things here. Can we reliably tell if a text was LLM generated? I just heard of a prof screening assignments for this, but not sure how that would work.

Of course there are people who will sell you a tool to do this. I sincerely doubt it's any good. But then again they can apparently fingerprint human authors fairly well using statistics from their writing, so what do I know.

Re: Updated practice for review articles and position papers in ArXiv CS category

#30

Post by DroneBetter » Sat, Nov 01, 2025, 3:29 PM UTC

arendtio wrote: post #11

I wonder why they can't facilitate LLMs in the review process (like fighting fire with fire). Are even the best models not capable enough, or are the costs too high?

the problem is generally the same as with generative adversarial networks; the capability to meaningfully detect some set of hallmarks of LLMs automatically is equivalent to the capability to avoid producing those, and LLMs are trained to predict (ie. be indistinguishable from) their source corpus of human-written text.

so the LLM detection problem is (theoretically) impossible for SOTA LLMs; in practice, it could be easier due to the RLHF stage inserting idiosyncrasies.