Two perspectives: Either (I) LLMs made survey papers irrelevant, or (II) LLMs killed a useful set of arXiv papers.
Updated practice for review articles and position papers in ArXiv CS category
151–160 of 250 posts
Re: Updated practice for review articles and position papers in ArXiv CS category
#152Earlier quoted context omitted.
The problem with an endorsement scheme is citation rings, ie groups of people who artificially inflate the perceived value of some line of work by citing each other. This is a problem even now, but it is kept in check by the fact that authors do not usually have any control over who reviews their paper. Indeed, in my area, reviews are double blind, and despite claims that “you can tell who wrote this anyway” research…
But you can choose to not trust people that are part of citation rings.
Here's a paper rejected for plagiarism. Why don't you click on the authors' names and look at their Google scholar pages... you can also look at their DBLP page and see who they publish with.
Also look how frequently they publish. Do you really think it's reasonable to produce a paper every week or two? Even if you have a team of grad students? I'll put it this way, I had a paper have difficulty getting through reviewer for "not enough experiments" when several of my experiments took weeks wall time to run and one took a month (could not run that a second time lol)
We don't do a great job at ousting frauds in science. It's actually difficult to do because science requires a lot of trust. We could alleviate some of these issues if we'd allow publication or some reward mechanism for replication, but the whole system is structured to reward "new" ideas. Utility isn't even that much of a factor in some areas. It's incredibly messy.
Most researchers are good actors. We all make mistakes and that's why it's hard to detect fraud. But there's also usually high reward for doing so. Though most of that reward is actually getting a stable job and the funding to do your research. Which is why you can see how it might be easy to slip into cheating a little here and there. There's ways to solve that that don't include punishing anyone...
Re: Updated practice for review articles and position papers in ArXiv CS category
#153Earlier quoted context omitted.
For position (opinion) or review (summarizing state of art and often laden with opinions on categories and future directions). LLMs would be happy to generate both these because they require zero technical contributions, working code, validated results, etc.
If you believe that, can you demonstrate how to generate a position or review paper using an LLM?
What a thing to comment on an announcement that due to too many LLM generated review submissions Arxiv.cs will officially no longer publish preprints of reviews.
Re: Updated practice for review articles and position papers in ArXiv CS category
#154I'm not sure this is the right way to handle it (I don't know what is) but arXiv.org has suffered from poor quality self-promotion papers in CS for a long time now. Years before llms.
How precisely does it "suffer" though? It's basically a way to disseminate results but carries no journalistic prestige in itself. It's a fun place to look now and then for new results, but just reading the "front page" of a category has always been a Caveat Emptor situation.
Because a large number of "preprints" that are really blog posts or advertisements for startup greatly increase the noise.
The idea is the site is for academic preprints. Academia has a long history of circulating preprints or manuscripts before the work is finished. There are many reasons for this, the primary one is that scientific and mathematical papers are often in the works for years before they get officially published. Preprints allow other academics in the know to be up to date on current results.
If the service is used heavily by non-academics to lend an aura of credibility to any kind of white paper then the service is less usable for its intended purpose.
It's similar to the use of question/answer sites like Quora to write blog posts and ads under questions like "Why is Foobar brand soap the right soap for your family?"
Re: Updated practice for review articles and position papers in ArXiv CS category
#155There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…
Sure, just as long as we don't blame LLMs. Blame people, bad actors, systems of incentives, the gods, the devils, but never broach the fault of LLMs and their wide spread abuse.
What would be the point of blaming LLMs? What would that accomplish? What does it even mean to blame LLMs?
LLMs are not submitting these papers on their own, people are. As far as I'm concerned, whatever blame exists rests on those people and the system that rewards them.
Re: Updated practice for review articles and position papers in ArXiv CS category
#156Great move by arXiv—clear standards for reviews and position papers are crucial in fast-moving areas like multi-agent systems and agentic LLMs. Requiring machine-readable metadata (type=review/position, inclusion criteria, benchmark coverage, code/data links) and consistent cross-listing (cs.AI/cs.MA) would help readers and tools filter claims, especially in distributed/parallel agentic AI where evaluation is fragile. A standardized “Survey”/“Position” tag plus a brief reproducibility checklist would set expectations without stifling early ideas.
Re: Updated practice for review articles and position papers in ArXiv CS category
#157Earlier quoted context omitted.
I had been kinda hoping for a web-of-trust system to replace peer review. Anyone can endorse an article. You can decide which endorsers you trust, and do some network math to find what you think is reading. With hashes and signatures and all that rot. Not as gate-keepy as journals and not as anarchic as purely open publishing. Should be cheap, too.
What prevents you from creating an island of fake endorsers?
A web of trust is transitive, meaning that the endorsers are known. It would be trivial to add negative weight to all endorsers of a known-fake paper, and only sightly less trivial to do the same for all endorsers of real papers artificially boosted by such a ring.
Re: Updated practice for review articles and position papers in ArXiv CS category
#158The number of content generators is now infinite but the number of content reviewers is the same.
Sorry folks but we lost.
Re: Updated practice for review articles and position papers in ArXiv CS category
#159Earlier quoted context omitted.
Sure, just as long as we don't blame LLMs. Blame people, bad actors, systems of incentives, the gods, the devils, but never broach the fault of LLMs and their wide spread abuse.
What would be the point of blaming LLMs? What would that accomplish? What does it even mean to blame LLMs? LLMs are not submitting these papers on their own, people are. As far as I'm concerned, whatever blame exists rests on those people and the system that rewards them.
Perhaps what is meant is "blame the development of LLMs." We don't "blame guns" for shootings, but certainly with reduced access to guns, shootings would be fewer.
Re: Updated practice for review articles and position papers in ArXiv CS category
#160Simple solution: criminalize posting AI generated publications IF NOT DISCLOSED CLEARLY. Lets say 50000€ fine, or 1 year in prison. :)
Literally everything will say AI generated to avoid potential liability. You'll have a "known to the state of California to cause cancer" situation.