Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updated practice for review articles and position papers in ArXiv CS category

blog.arxiv.org

Post reply on HN
Print view View original

221–230 of 250 posts

Re: Updated practice for review articles and position papers in ArXiv CS category

#221

Post by LunaSea »

noobermin wrote: post #211
miki123211 wrote: post #194

Earlier quoted context omitted.

LLMs are tools that make it easier to hack incentives, but you still need a person to decide that they'll use an LLM t do so. Blaming LLMs is unproductive. They are not going anywhere (especially since open source LLMs are so good.) If we want to achieve real change, we need to accept that they exist, understand how that changes the scientific landscape and our options to go from here.

everyone keeps claiming "they're here to stay" as if it's gospel. this constant drumbeat is rather tiresome and without much hard evidence.

Genuinely curious, did we ever manage to ban a piece of technology worldwide and effectively?

Re: Updated practice for review articles and position papers in ArXiv CS category

#222

Post by andrybak »

LunaSea wrote: post #221
noobermin wrote: post #211

Earlier quoted context omitted.

everyone keeps claiming "they're here to stay" as if it's gospel. this constant drumbeat is rather tiresome and without much hard evidence.

Genuinely curious, did we ever manage to ban a piece of technology worldwide and effectively?

Do chlorofluorocarbons (CFCs) mostly banned by the Montreal Protocol count?

Re: Updated practice for review articles and position papers in ArXiv CS category

#223

Post by naasking »

me_again wrote: post #185
naasking wrote: post #131

Earlier quoted context omitted.

So what? People are experimenting with novel tools for review and publication. These restrictions are dumb, people can just ignore reviews and position papers if they start proving to be less useful, and the good ones will eventually spread through word of mouth, just like arxiv has always worked.

ArXiv has always had a moderation step. The moderators are unable to keep up with the volume of submissions. Accepting these reviews without moderation would be a change to current process, not "just like arXiv has always worked"

Setting aside the wisdom of moderation, instead of banning AI, use it to accelerate review.

Re: Updated practice for review articles and position papers in ArXiv CS category

#224

Post by wizzwizz4 »

alfalfasprout wrote: post #201
hammock wrote: post #199

Earlier quoted context omitted.

> There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, I heard someone say something similar about the “homeless industrial complex” on a podcast recently. I think it was San Francisco that pays NGOs funds for homeless aid based on how many homeless people they ser…

It's a metric attribution problem. The real metric should be reduction in homeless, for example (though even that can be gamed through bussing them out, etc-- tactics that unfortunately other cities have adopted). But attributing that to a single NGO is tough. Ditto for views, etc. Really what you care about as eg; youtube is conversions for the products that are advertised. Not impressions. But there's an attributio…

Define the metric as "people helped": then bussing them out to abandon them somewhere else isn't a solution, because the adjudicators can go "yes, you made the number go down, but you did so by decoupling the metric from what it was supposed to measure, so we're not rewarding you for it".

Re: Updated practice for review articles and position papers in ArXiv CS category

#225

Post by jruohonen »

A very weird move. They are now taking a stance on what science is supposed to be.

As someone commented, due to the increasing volume, we would actually need and benefit from more reviews -- with a fixed cycle preferably, and I do not mean LLM slop but SLRs. And in contrary to someone's post, it is actually nice to read things from the industry, and I would actually want that more.

And not only are they taking a stance on science but they have also this allegation:

"Please note: the review conducted at conference workshops generally does not meet the same standard of rigor of traditional peer review and is not enough to have your review article or position paper accepted to arXiv."

In fact -- and supposedly related to the peer review crisis, the situation is exactly the opposite. That is, reviews are usually today of much higher quality at specialized workshops organized by experts in a particular, often niche area.

Maybe arXiv people should visit PubPeer once in a while to see what kind of fraud is going on with conferences (i.e., not workshops and usually not review papers) and their proceedings published by all notable CS publishers? The same goes for journals.

Re: Updated practice for review articles and position papers in ArXiv CS category

#226

Post by xhkkffbf »

wizzwizz4 wrote: post #224
alfalfasprout wrote: post #201

Earlier quoted context omitted.

It's a metric attribution problem. The real metric should be reduction in homeless, for example (though even that can be gamed through bussing them out, etc-- tactics that unfortunately other cities have adopted). But attributing that to a single NGO is tough. Ditto for views, etc. Really what you care about as eg; youtube is conversions for the products that are advertised. Not impressions. But there's an attributio…

Define the metric as "people helped": then bussing them out to abandon them somewhere else isn't a solution, because the adjudicators can go "yes, you made the number go down, but you did so by decoupling the metric from what it was supposed to measure, so we're not rewarding you for it".

And then some will wander back closing the loop and preserving jobs.

Re: Updated practice for review articles and position papers in ArXiv CS category

#227

Post by pjdesno »

_jsmh wrote: post #173
efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

What would a system that rewards people for quality rather than volume look like? How would an online world that is optimized for humans, not algorithms, look like? Should content creators get paid?

> What would a system that rewards people for quality rather than volume look like?

Hiring and tenure review based on a candidate’s selected 5 best papers.

Already standard practice at a few enlightened places, I think. (of course this also probably increases the review workload for top venues)

To a lesser extent, bean-counting metrics like citations and h-index are an attempt to quantify non-volume-based metrics. (for non-academics, h-index is the largest N such that your N-th most cited paper has >= N citations)

Note that most approaches like this have evolved to counter “salami-slicing”, where you divide your work into “minimum publishable units”. LLMs are a different threat - from my selfish point of view, one of the biggest risks is that it takes less time to write a bogus paper with an LLM than it does for a single reviewer to review it. That threatens to upend the entire peer reviewing process.

Re: Updated practice for review articles and position papers in ArXiv CS category

#228

Post by SOLAR_FIELDS »

ssivark wrote: post #207
djeastm wrote: post #204

Earlier quoted context omitted.

I don't really buy it. Are we to believe they go out of their way to keep people homeless? Does the same logic apply to doctors keeping people sick?

ICYMI, this drew a lot of attention a few years ago. https://www.cnbc.com/2018/04/11/goldman-asks-is-curing-patie...

This could literally be an Onion headline

Re: Updated practice for review articles and position papers in ArXiv CS category

#229

Post by SOLAR_FIELDS »

wizzwizz4 wrote: post #224
alfalfasprout wrote: post #201

Earlier quoted context omitted.

It's a metric attribution problem. The real metric should be reduction in homeless, for example (though even that can be gamed through bussing them out, etc-- tactics that unfortunately other cities have adopted). But attributing that to a single NGO is tough. Ditto for views, etc. Really what you care about as eg; youtube is conversions for the products that are advertised. Not impressions. But there's an attributio…

Define the metric as "people helped": then bussing them out to abandon them somewhere else isn't a solution, because the adjudicators can go "yes, you made the number go down, but you did so by decoupling the metric from what it was supposed to measure, so we're not rewarding you for it".

My spouse works in the homelessness field and the correct metric to follow is number of homeless given housing. It’s the “housing first” approach. Harder to game counting amount of people directly placed into homes - someone is paying rent and maintaining a trackable occupied space that you can verify that the client is actually utilizing - and this approach cannot be gamed by “bus them somewhere else”

What many people don’t realize is just how many normal life hurdles are significantly easier to overcome with a stable housing environment, even if the client is willing and available to work. Employment, for example, has several precursors that you need. Often you need an address. You need an ID. For that you need a birth certificate. To get the birth certificate you need to have the resources and know how to contact the correct agency. All of these things are much harder to achieve without a stable housing environment for the client.

Re: Updated practice for review articles and position papers in ArXiv CS category

#230

Post by wizzwizz4 »

naasking wrote: post #223
me_again wrote: post #185

Earlier quoted context omitted.

ArXiv has always had a moderation step. The moderators are unable to keep up with the volume of submissions. Accepting these reviews without moderation would be a change to current process, not "just like arXiv has always worked"

Setting aside the wisdom of moderation, instead of banning AI, use it to accelerate review.

Unfortunately, (this kind of) AI doesn't accelerate review. (That's before you get into the ease of producing adversarial inputs: a moderation system not susceptible to these could be wired up backwards as a generation system that produces worthwhile research output, and we don't have one of those.)
Post reply on HN