The Tragedy of the Commons, updated for LLMs. Part #975 in a continuing series.
These things will ruin everything good, and that is before we even start talking about audio or video.
Updated practice for review articles and position papers in ArXiv CS category
41–50 of 250 posts
Re: Updated practice for review articles and position papers in ArXiv CS category
#42Maybe it's time for a reputation system. E.g. every author publishes a public PGP key along with their work. Not sure about the details but this is about CS, so I'm sure they will figure something out.
I had been kinda hoping for a web-of-trust system to replace peer review. Anyone can endorse an article. You can decide which endorsers you trust, and do some network math to find what you think is reading. With hashes and signatures and all that rot.
Not as gate-keepy as journals and not as anarchic as purely open publishing. Should be cheap, too.
Re: Updated practice for review articles and position papers in ArXiv CS category
#43Earlier quoted context omitted.
Their name, orcid, and email isn't enough?
You can’t get an arXiv account without a referral anyway. Edit: For clarification I’m agreeing with OP
Not quite true. If you've got an email associated with a known organization you can submit.
Which includes some very large ones like @google.com
Re: Updated practice for review articles and position papers in ArXiv CS category
#44Simple solution: criminalize posting AI generated publications IF NOT DISCLOSED CLEARLY.
Lets say 50000€ fine, or 1 year in prison. :)
Re: Updated practice for review articles and position papers in ArXiv CS category
#45I'm not sure this is the right way to handle it (I don't know what is) but arXiv.org has suffered from poor quality self-promotion papers in CS for a long time now. Years before llms.
How precisely does it "suffer" though? It's basically a way to disseminate results but carries no journalistic prestige in itself. It's a fun place to look now and then for new results, but just reading the "front page" of a category has always been a Caveat Emptor situation.
Re: Updated practice for review articles and position papers in ArXiv CS category
#46it's clearly not sutainable to have the main website hosting CS articles not having any reviews or restrictions. (Except for the initial invite system) There were 26k submission in october: https://arxiv.org/stats/monthly_submissions Asking for a small amount of money would probably help. Issue with requiring peer reviewed journals or conferences is the severe lag, takes a long time and part of the advantage of arxiv…
I like this idea. A small contribution would be a good filter. Looking at the stats it’s quite crazy. Didn’t know that we could access to this data. Thanks for sharing.
Re: Updated practice for review articles and position papers in ArXiv CS category
#47i would like to understand what people get, or think they get, out of putting a completely AI-generated survey paper on arXiv.
Even if AI writes the paper for you, it's still kind of a pain in the ass to go through the submission process, get the LaTeX to compile on their servers, etc., there is a small cost to you. Why do this?
Re: Updated practice for review articles and position papers in ArXiv CS category
#48Earlier quoted context omitted.
A good review paper is infinitely better than an llm managing to find a few papers and making a summary. A knowledgeable researcher knows which papers are outdated and can make a trustworthy review paper, an LLM can't easily do that yet
Ok I take your point. However, it is possible to generate a middling review paper combining ai generated slop and edits. Maybe we would be tricked by it in certain circumstances. I don't mean to imply these outputs are something I would value reading. I am just arguing in favour of the proposed approach of arXiv.
> it is possible to generate a middling review paper combining ai generated slop and edits
If you’re an expert. If you’re not, you’ll publish, best case, bullshit. (Worst case lies.)
Re: Updated practice for review articles and position papers in ArXiv CS category
#49So what they no longer accept is preprints (or rejects…) It’s of course a pretty big deal given that arXiv is all about preprints. And an accepted journal paper presumably cannot be submitted to arXiv anyway unless it’s an open journal.
Isnt arxiv also a likely LLM traing ground?
Re: Updated practice for review articles and position papers in ArXiv CS category
#50Earlier quoted context omitted.
It is actually great because it shows how well it works as a system. Screening is really important to keep preprint quality high enough to then implement cool ideas like random peer review/automated reviews etc
> we are developing a whole new method to do peer review What’s the new method?
I mean generally working towards changing how peer review works.
For example: https://prereview.org/en-us
Anecdotally, a lot of researchers will run their paper pdfs through an AI iteration or two during drafting which also (kinda but not really) counts as a self-review. Although that is not comparable to peer review ofc.