This should honestly have been implemented a long time ago. Much of academia is pressured to churn out papers month after month as academia is prioritizing volume over quality or impact.
Updated practice for review articles and position papers in ArXiv CS category
111–120 of 250 posts
Re: Updated practice for review articles and position papers in ArXiv CS category
#112The HN submission title is incorrect. > Before being considered for submission to arXiv’s CS category, review articles and position papers must now be accepted at a journal or a conference and complete successful peer review. Edit: original title was "arXiv No Longer Accepts Computer Science Position or Review Papers Due to LLMs"
I don’t know about this. From a pure entertainment standpoint, we may be denying ourselves a world of hilarity. LLMs + “You know Peter, I’m something of a research myself” delusions. I’d pay for this so long as people are very serious about the delusion.
Re: Updated practice for review articles and position papers in ArXiv CS category
#113Maybe it's time for a reputation system. E.g. every author publishes a public PGP key along with their work. Not sure about the details but this is about CS, so I'm sure they will figure something out.
I got that suggestion recently talking to a colleague from a prestigious university. Her suggestion was simple: Kick out all non-ivy league and most international researchers. Then you have a working reputation system. Make of that what you will ...
all non-ivy league researchers? that seems a little harsh IMO. i've read some amazing papers from T50 or even some T100 universities.
Re: Updated practice for review articles and position papers in ArXiv CS category
#114I suspect that LLMs are better at classifying novel vs junk papers than they are at creating novel papers themselves.
If so, I think the solution is obvious.
(But I remind myself that all complex problems have a simple solution that is wrong.)
Re: Updated practice for review articles and position papers in ArXiv CS category
#115So what they no longer accept is preprints (or rejects…) It’s of course a pretty big deal given that arXiv is all about preprints. And an accepted journal paper presumably cannot be submitted to arXiv anyway unless it’s an open journal.
> And an accepted journal paper presumably cannot be submitted to arXiv anyway unless it’s an open journal.
You cannot upload the journal’s version, but you can upload the text as accepted (so, the same content minus the formatting).
Re: Updated practice for review articles and position papers in ArXiv CS category
#116Why not just reject papers authored by LLMs and ban accounts that are caught? arXiv’s management has become really questionable lately, it’s like they’re trying to become a prestigious journal and are becoming the problem they were trying to solve in the first place
> Why not just reject papers authored by LLMs and ban accounts that are caught? Are you saying that there's an automated method for reliably verifying that something was created by an LLM?
If there wasn’t, then how do they know LLMs are the problem?
Re: Updated practice for review articles and position papers in ArXiv CS category
#117I suspect that LLMs are better at classifying novel vs junk papers than they are at creating novel papers themselves. If so, I think the solution is obvious. (But I remind myself that all complex problems have a simple solution that is wrong.)
Verification via LLM tends to break under quite small optimization pressure. For example I did RL to improve against one of the sota models from one generation ago, and the (quite weak) learner model found out that it could emit a few nonsense words to get the max score.
That's without even being able to backprop through the annotator, and also with me actively trying to avoid reward hacking. If arxiv used an open model for review, it would be trivial for people to insert a few grammatical mistakes which cause them to receive max points.
Re: Updated practice for review articles and position papers in ArXiv CS category
#118I suspect that LLMs are better at classifying novel vs junk papers than they are at creating novel papers themselves. If so, I think the solution is obvious. (But I remind myself that all complex problems have a simple solution that is wrong.)
> I suspect that LLMs are better at classifying novel vs junk papers than they are at creating novel papers themselves.
Doubt
LLMs are experts in generating junk. And generally terrible at anything novel. Classifying novel vs junk is a much harder problem.
Re: Updated practice for review articles and position papers in ArXiv CS category
#119Earlier quoted context omitted.
I didn't agree with this idea, but then I looked at how much HN karma you have and now I think that maybe this is a good idea.
Ignoring the actual proposal or user, just looking at karma is probably a pretty terrible metric. High karma accounts tend to just interact more frequently, for long periods of time. Often with less nuanced takes, that just play into what is likely to be popular within a thread. Having a Userscript that just places the karma and comment count next to a username is pretty eye opening.
I have a userscript to actually hide my own karma because I always think it is useless but your point is good actually. But also I think that karma/comment ratio is better than absolute karma. It has its own problems but it is just better. And I would ask if you can share the userscript.
And to bring this back to the original arxiv topic. I think reputation system is going to face problems with some people outside CS lack of enough technical abilities. It also introduce biases in that you would endorse people who you like for other reasons. Actually some of the problems are solved and you would need careful proposal. But the change for publishing scheme needs push from institutions and funding agencies. Authors don't oppose changes but you have a lobby of the parasitic publishing cartel that will oppose these changes.
Re: Updated practice for review articles and position papers in ArXiv CS category
#120Isn’t github the normal way of publishing now for cs ?
Does Google Scholar index it?