Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updated practice for review articles and position papers in ArXiv CS category

blog.arxiv.org

Post reply on HN
Print view View original

141–150 of 250 posts

Re: Updated practice for review articles and position papers in ArXiv CS category

#141

Post by ec109685 »

DalasNoin wrote: post #10

it's clearly not sutainable to have the main website hosting CS articles not having any reviews or restrictions. (Except for the initial invite system) There were 26k submission in october: https://arxiv.org/stats/monthly_submissions Asking for a small amount of money would probably help. Issue with requiring peer reviewed journals or conferences is the severe lag, takes a long time and part of the advantage of arxiv…

It’s not a money issue. People publish these papers to get jobs, into schools, visa’s and whatnot. Way more than $30 in value from being “published”.

Re: Updated practice for review articles and position papers in ArXiv CS category

#142

Post by ec109685 »

currymj wrote: post #47

i would like to understand what people get, or think they get, out of putting a completely AI-generated survey paper on arXiv. Even if AI writes the paper for you, it's still kind of a pain in the ass to go through the submission process, get the LaTeX to compile on their servers, etc., there is a small cost to you. Why do this?

Published papers are part of the EB-1 visa rubric so huge value in getting your content into these indexes:

"One specific criterion is the ‘authorship of scholarly articles in professional or major trade publications or other major media’. The quality and reputation of the publication outlet (e.g., impact factor of a journal, editorial review process) are important factors in the evaluation”

Re: Updated practice for review articles and position papers in ArXiv CS category

#143

Post by phi-go »

dmoy wrote: post #80
geysersam wrote: post #74

Earlier quoted context omitted.

But you can choose to not trust people that are part of citation rings.

It is a non trivial problem to do just that. It's related to the same problems you have with e.g. Sybil attacks: https://en.wikipedia.org/wiki/Sybil_attack I'm not saying it wouldn't be worthwhile to try, just that I expect there to be a lot of very difficult problems to solve there.

With peer review you do not even have a choice as to which reviewers to trust as it is all homogenized by acceptance or not. This is marginally better if reviews are published.

That is to say I also think it would be worthwhile to try.

Re: Updated practice for review articles and position papers in ArXiv CS category

#144

Post by bjourne »

jvanderbot wrote: post #19
Sharlin wrote: post #5

So what they no longer accept is preprints (or rejects…) It’s of course a pretty big deal given that arXiv is all about preprints. And an accepted journal paper presumably cannot be submitted to arXiv anyway unless it’s an open journal.

For position (opinion) or review (summarizing state of art and often laden with opinions on categories and future directions). LLMs would be happy to generate both these because they require zero technical contributions, working code, validated results, etc.

If you believe that, can you demonstrate how to generate a position or review paper using an LLM?

Re: Updated practice for review articles and position papers in ArXiv CS category

#145

Post by [deleted] »

aoki wrote: post #130
ivape wrote: post #112

Earlier quoted context omitted.

I don’t know about this. From a pure entertainment standpoint, we may be denying ourselves a world of hilarity. LLMs + “You know Peter, I’m something of a research myself” delusions. I’d pay for this so long as people are very serious about the delusion.

That’s viXra

[deleted]

Re: Updated practice for review articles and position papers in ArXiv CS category

#146

Post by arendtio »

efavdb wrote: post #17
arendtio wrote: post #11

I wonder why they can't facilitate LLMs in the review process (like fighting fire with fire). Are even the best models not capable enough, or are the costs too high?

Curious for the state on things here. Can we reliably tell if a text was LLM generated? I just heard of a prof screening assignments for this, but not sure how that would work.

Well, I think it depends on how much effort the 'writer' is going to invest. If the writer simply tells the LLM to write something, you can be fairly certain it can be identified. However, I am not sure if the 'writer' provides extensive style instructions (e.g., earlier works by the same author).

Anecdotal: A few weeks ago, I came across a story on HN where many commenters immediately recognized that an LLM had written the article, and the author had actually released his prompts and iterations. So it was not a one-shot prompt but more like 10 iterations, and still, many people saw that an LLM wrote it.

Re: Updated practice for review articles and position papers in ArXiv CS category

#147

Post by uniqueuid »

Ekaros wrote: post #129
uniqueuid wrote: post #79

Earlier quoted context omitted.

I got that suggestion recently talking to a colleague from a prestigious university. Her suggestion was simple: Kick out all non-ivy league and most international researchers. Then you have a working reputation system. Make of that what you will ...

Maybe there should be some type of strike rules. Say 3 bad articles from any institution and they get 10 year ban. Whatever their prestige or monetary value is. You let people under your name to release bad articles you are out for a while. Treat everyone equally. After 10 years of only quality you get chance to get back. Before that though luck.

I'm not sure everyone got my hint that the proposal is obviously very bad,

(1) because ivy league also produces a lot of work that's not so great (i.e. wrong (looking at you, Ariely) or un-ambitious) and

(2) because from time to time, some really important work comes out of surprising places.

I don't think we have a good verdict on the Orthega hypothesis yet, but I'm not a professional meta scientist.

That said, your proposal seems like a really good idea, I like it! Except I'd apply it to individuals and/or labs.

Re: Updated practice for review articles and position papers in ArXiv CS category

#148

Post by godelski »

efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

  > rewarding people for the volume ... rather than the quality.
I suspect this is a major part of the appeal of LLMs themselves. They produce lines very fast so it appears as if work is being done fast. But that's very hard to know because number of lines is actually a zero signal in code quality or even a commit. Which it's a bit insane already that we use number of lines and commits as measures in the first place. They're trivial to hack. You even just reward that annoying dude who keeps changing the file so the diff is the entire file and not the 3 lines they edited...

I've been thinking we're living in "Goodhart's Hell". Where metric hacking has become the intent. That we've decided metrics are all that matter and are perfectly aligned with our goals.

But hey, who am I to critique. I'm just a math nerd. I don't run a multi trillion dollar business that lays off tons of workers because the current ones are so productive due to AI that they created one of the largest outages in history of their platform (and you don't even know which of the two I'm referencing!). Maybe when I run a multi trillion dollar business I'll have the right to an opinion about data.

Re: Updated practice for review articles and position papers in ArXiv CS category

#149

Post by bc569a80a344f9c »

hermannj314 wrote: post #91
amelius wrote: post #8

Maybe it's time for a reputation system. E.g. every author publishes a public PGP key along with their work. Not sure about the details but this is about CS, so I'm sure they will figure something out.

I didn't agree with this idea, but then I looked at how much HN karma you have and now I think that maybe this is a good idea.

I think it’s lovely that at the time of my reply, everyone seems to be taking your comment at face value instead of for the meta-commentary on “people upvoting content” you’re making by comparing HN karma to endorsement of papers via PGP signatures.

Re: Updated practice for review articles and position papers in ArXiv CS category

#150

Post by arendtio »

DroneBetter wrote: post #30
arendtio wrote: post #11

I wonder why they can't facilitate LLMs in the review process (like fighting fire with fire). Are even the best models not capable enough, or are the costs too high?

the problem is generally the same as with generative adversarial networks; the capability to meaningfully detect some set of hallmarks of LLMs automatically is equivalent to the capability to avoid producing those, and LLMs are trained to predict (ie. be indistinguishable from) their source corpus of human-written text. so the LLM detection problem is (theoretically) impossible for SOTA LLMs; in practice, it could be…

Sure, having a 100% reliable system is impossible as you have laid out. However, if I understand the announcement correctly, this is about volume, and I wonder if you could have a tool flag articles that show obvious signs of LLM usage.
Post reply on HN