Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updated practice for review articles and position papers in ArXiv CS category

blog.arxiv.org

Post reply on HN
Print view View original

191–200 of 250 posts

Re: Updated practice for review articles and position papers in ArXiv CS category

#191

Post by vladms »

_jsmh wrote: post #173
efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

What would a system that rewards people for quality rather than volume look like? How would an online world that is optimized for humans, not algorithms, look like? Should content creators get paid?

> Should content creators get paid?

Everybody "creates content" (like me when I take a picture of beautiful sunset).

There is no such thing as "quality". There is quality for me and quality for you. That is part of the problem, we can't just relate to some external, predefined scale. We (the sum of people) are the approximate, chaotic, inefficient scale.

Be my guest to propose a "perfect system", but - just in case there is no such system - we should make sure each of us "rewards" what we find of quality (being people or content creators), and hope it will prevail. Seemed to have worked so far.

Re: Updated practice for review articles and position papers in ArXiv CS category

#192

Post by bulubulu »

physarum_salad wrote: post #14

The review paper is dead... so this is a good development. Like you can generate these things in a couple of iterations with AI and minor edits. Preprint servers could be dealing with 1000s of review/position papers over short periods of time and then this wastes precious screening work hours. It is a bit different in other fields where interpretations or know-how might be communicated in a review paper format that i…

Review papers are summarizations to recent updates in the field that deserve fellow researchers' attention. Such works should be done annually or at most quarterly in my opinion, to include only time-tested results. If hundreds of review papers are published every month, I am afraid that their quality in terms of paper selection and innovative interpretation/direction will not be much higher than the content generated by LLM, even if written word-to-word by a real scientist.

LLMs are good at plainly summarizing from the public knowledge base. Scientists should invest their time in contributing new knowledge to public base instead of doing the summarization.

Re: Updated practice for review articles and position papers in ArXiv CS category

#193

Post by anonym29 »

noobermin wrote: post #133
efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

Sure, just as long as we don't blame LLMs. Blame people, bad actors, systems of incentives, the gods, the devils, but never broach the fault of LLMs and their wide spread abuse.

This was a problem before LLMs and it would remain a problem if you could magically make all of them disappear.

LLMs are not the root of the problem here.

Re: Updated practice for review articles and position papers in ArXiv CS category

#194

Post by miki123211 »

noobermin wrote: post #133
efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

Sure, just as long as we don't blame LLMs. Blame people, bad actors, systems of incentives, the gods, the devils, but never broach the fault of LLMs and their wide spread abuse.

LLMs are tools that make it easier to hack incentives, but you still need a person to decide that they'll use an LLM t do so.

Blaming LLMs is unproductive. They are not going anywhere (especially since open source LLMs are so good.)

If we want to achieve real change, we need to accept that they exist, understand how that changes the scientific landscape and our options to go from here.

Re: Updated practice for review articles and position papers in ArXiv CS category

#195

Post by vladms »

fn-mote wrote: post #126
hermannj314 wrote: post #91

Earlier quoted context omitted.

I didn't agree with this idea, but then I looked at how much HN karma you have and now I think that maybe this is a good idea.

I would be much happer if you explained your _reasons_ for disagreeing or your _reasons_ for agreeing. I don't think publishing a PGP key with your work does anything. There's no problem identifying the author of the work. The problem is identifying _untrustworthy_ authors. Especially in the face of many other participants in the system claiming the work is trusted. As I understand it, the current system (in some fie…

> The problem is identifying _untrustworthy_ authors.

Is it though? Should we care about authors or about the work? Yes, many experiments are hard to reproduce, but isn't that something we should work towards, rather than just "trust" someone. People change. People do mistakes. I think more open data, open access, open tools, will solve a lot, but my guess is that generally people do not like that because it can show their weaknesses - even if they are well intentioned.

Re: Updated practice for review articles and position papers in ArXiv CS category

#196

Post by nsagent »

nandomrumber wrote: post #162
jsrozner wrote: post #159

Earlier quoted context omitted.

Perhaps what is meant is "blame the development of LLMs." We don't "blame guns" for shootings, but certainly with reduced access to guns, shootings would be fewer.

Guns have absolutely nothing to do with access to guns. Guns are entirely inert objects, devoid of either free will nor volition, they have no rights and no responsibilities. LLMs likewise.

  To every man is given the key to the gates of heaven. The same key opens the gates of hell.
-Richard Feynman

https://www.goodreads.com/quotes/421467-to-every-man-is-give...

https://calteches.library.caltech.edu/1575/1/Science.pdf

Re: Updated practice for review articles and position papers in ArXiv CS category

#197

Post by xandrius »

noobermin wrote: post #133
efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

Sure, just as long as we don't blame LLMs. Blame people, bad actors, systems of incentives, the gods, the devils, but never broach the fault of LLMs and their wide spread abuse.

I blame keyboards, without them there wouldn't be these problems.

Re: Updated practice for review articles and position papers in ArXiv CS category

#198

Post by amelius »

amelius wrote: post #136
SyrupThinker wrote: post #103

Earlier quoted context omitted.

Ignoring the actual proposal or user, just looking at karma is probably a pretty terrible metric. High karma accounts tend to just interact more frequently, for long periods of time. Often with less nuanced takes, that just play into what is likely to be popular within a thread. Having a Userscript that just places the karma and comment count next to a username is pretty eye opening.

Yes, HN should probably publish karma divided by #comments. Or at least show both numbers.

(an added complication is that posting articles also increases karma)

Re: Updated practice for review articles and position papers in ArXiv CS category

#199

Post by hammock »

efitz wrote: post #89

There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system, meeting the minimum quality bar, while taking the least effort to create the most papers and thereby receive the greatest reward. Similarly, if you reward content creators based on views, you will get view maximi…

> There is a general problem with rewarding people for the volume of stuff they create, rather than the quality. If you incentivize researchers to publish papers, individuals will find ways to game the system,

I heard someone say something similar about the “homeless industrial complex” on a podcast recently. I think it was San Francisco that pays NGOs funds for homeless aid based on how many homeless people they serve. So the incentive is to keep as many homeless around as possible, for as long as possible.

Re: Updated practice for review articles and position papers in ArXiv CS category

#200

Post by nickpsecurity »

slashdave wrote: post #178
nickpsecurity wrote: post #125

Earlier quoted context omitted.

I'll add the amount should be enough to cover at least a cursory review. A full review would be better. I just don't want to price out small players. The papers could also be categorized as unreviewed, quick check, fully reviewed, or fully reproduced. They could pay for this to be done or verified. Then, we have a reputational problem to deal with on the reviewer side.

> I'll add the amount should be enough to cover at least a cursory review. You might be vastly underestimating the cost of such a feature

I'm assuming it cost somewhere between no review and a thorough one. Past that, I assume nothing. Pay reviewers per review or per hour like other consultants. Groups like Arxiv would, for a smaller fee, verify the reviewer's credentials and that the review happened.

That's if anyone wants the publishing to be closer to thr scientific method. Arxiv themselves might not attempt all of that. We can still hope for volunteers to review papers in a field with little, peer review. I just don't think we can call most of that science anymore.

Post reply on HN