Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updated practice for review articles and position papers in ArXiv CS category

blog.arxiv.org

Post reply on HN
Print view View original

201–210 of 250 posts

Re: Updated practice for review articles and position papers in ArXiv CS category

#201

Post by alfalfasprout »

hammock wrote: post #199
efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

> There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, I heard someone say something similar about the “homeless industrial complex” on a podcast recently. I think it was San Francisco that pays NGOs funds for homeless aid based on how many homeless people they ser…

It's a metric attribution problem. The real metric should be reduction in homeless, for example (though even that can be gamed through bussing them out, etc-- tactics that unfortunately other cities have adopted). But attributing that to a single NGO is tough.

Ditto for views, etc. Really what you care about as eg; youtube is conversions for the products that are advertised. Not impressions. But there's an attribution problem there.

Re: Updated practice for review articles and position papers in ArXiv CS category

#202

Post by bjourne »

SiempreViernes wrote: post #153
bjourne wrote: post #144

Earlier quoted context omitted.

If you believe that, can you demonstrate how to generate a position or review paper using an LLM?

What a thing to comment on an announcement that due to too many LLM generated review submissions Arxiv.cs will officially no longer publish preprints of reviews.

Not what the announcement says. And if you're so sure it's possible, show us how it's done.

Re: Updated practice for review articles and position papers in ArXiv CS category

#203

Post by bjourne »

dredmorbius wrote: post #170
bjourne wrote: post #144

Earlier quoted context omitted.

If you believe that, can you demonstrate how to generate a position or review paper using an LLM?

[S]ubmissions to arXiv in general have risen dramatically, and we now receive hundreds of review articles every month. The advent of large language models have made this type of content relatively easy to churn out on demand, and the majority of the review articles we receive are little more than annotated bibliographies, with no substantial discussion of open research issues. arXiv believes that there are position p…

"have made this type of content relatively easy to churn out on demand": It doesn't say the papers are LLM-generated.

Re: Updated practice for review articles and position papers in ArXiv CS category

#204

Post by djeastm »

hammock wrote: post #199
efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

> There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, I heard someone say something similar about the “homeless industrial complex” on a podcast recently. I think it was San Francisco that pays NGOs funds for homeless aid based on how many homeless people they ser…

I don't really buy it. Are we to believe they go out of their way to keep people homeless? Does the same logic apply to doctors keeping people sick?

Re: Updated practice for review articles and position papers in ArXiv CS category

#206

Post by trostaft »

bee_rider wrote: post #38
physarum_salad wrote: post #14

The review paper is dead... so this is a good development. Like you can generate these things in a couple of iterations with AI and minor edits. Preprint servers could be dealing with 1000s of review/position papers over short periods of time and then this wastes precious screening work hours. It is a bit different in other fields where interpretations or know-how might be communicated in a review paper format that i…

What are review papers for anyway? I think they are either for 1) new grad students to end up with something nice to publish after reviewing the literature or, 2) older professors to write a big overview of everything that happened in their field as sort of a “bible” that can get you up to speed The former is useful as a social construct; I mean, hey, new grad students, don’t skimp on your literature review. Finding…

I've found (good) review papers invaluable as an academic. They're really useful as a fast ladder to getting up to speed in a new area. Usually they have a great literature review (with the important papers to read afterward), a curated list of results important to understand, and good intuition about how to reason. It's a compactification of what I would have to otherwise gain by working in an area for years. No replacement for it, of course, but does make it easier attain.

I don't understand the appeal of an (majorly-)LLM generated review paper. A good review paper is a hard task to write well, and frankly the only good ones I've read have come from authors who are at apex of their field (and are, in particular, strong writers). The 'lossy search' of an LLM is probably an outstanding tool for _refining_ a review paper, but for fully generating it? At least not with current LLMs.

Re: Updated practice for review articles and position papers in ArXiv CS category

#207

Post by ssivark »

djeastm wrote: post #204
hammock wrote: post #199

Earlier quoted context omitted.

> There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, I heard someone say something similar about the “homeless industrial complex” on a podcast recently. I think it was San Francisco that pays NGOs funds for homeless aid based on how many homeless people they ser…

I don't really buy it. Are we to believe they go out of their way to keep people homeless? Does the same logic apply to doctors keeping people sick?

ICYMI, this drew a lot of attention a few years ago.

https://www.cnbc.com/2018/04/11/goldman-asks-is-curing-patie...

Re: Updated practice for review articles and position papers in ArXiv CS category

#208

Post by [deleted] »

bjourne wrote: post #202
SiempreViernes wrote: post #153

Earlier quoted context omitted.

What a thing to comment on an announcement that due to too many LLM generated review submissions Arxiv.cs will officially no longer publish preprints of reviews.

Not what the announcement says. And if you're so sure it's possible, show us how it's done.

[deleted]

Re: Updated practice for review articles and position papers in ArXiv CS category

#209

Post by RobotToaster »

efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

See Goodhart's law: "When a measure becomes a target, it ceases to be a good measure"

Re: Updated practice for review articles and position papers in ArXiv CS category

#210

Post by MangoToupe »

_jsmh wrote: post #173
efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

What would a system that rewards people for quality rather than volume look like? How would an online world that is optimized for humans, not algorithms, look like? Should content creators get paid?

Crazily, I think the easiest way is to remove any and all incentives, awards, finite funding, and allegedly merit-based positions. Allow anyone who wants to research to research. Natural recognition of peers seems to be the only way to my thinking. Of course this relies on a post-scarcity society so short of actually achieving communism we'll likely never see it happen.
Post reply on HN