Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New arXiv policy: 1-year ban for hallucinated references

twitter.com

Post reply on HN
Print view View original

61–70 of 242 posts

Re: New arXiv policy: 1-year ban for hallucinated references

#61

Post by toast0 »

dataflow wrote: post #41
toast0 wrote: post #33

Earlier quoted context omitted.

> It's just one darn hallucinated citation for heaven's sake, not fraud or something. It is fraud. > It doesn't account for the substance or quality of their work at all. References are part of the work. If you're making up the references, what else are you making up? > People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's ability to progress…

> It is fraud. No, it is emphatically not. Fraud requires intent to deceive. > A one year ban is not permanent. ...what text are you reading? Nobody was calling the one-year ban permanent, or even against it. I was literally in favor of it in my comment. I explicitly said it is already plenty sufficient. What I said is there's no need to go beyond that. My entire gripe was that they very much are going beyond that wi…

> followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue"?

This part seemed reasonable too. I'm not in academia, but my understanding is most people writing papers intend for them to be accepted by reputable peer-reviewed venues, but post to arXiv because those venues don't always allow for simple distribution.

If your papers aren't going to be accepted at reputable venues and you posted slop to arXiv before (and they noticed it!), seems reasonable that they only want reputable stuff from you in the future?

Re: New arXiv policy: 1-year ban for hallucinated references

#63

Post by toast0 »

dataflow wrote: post #57
LPisGood wrote: post #53

Earlier quoted context omitted.

Fraud requires intent to deceive _or_ reckless disregard, sometimes called, “conscious indifference” for the veracity of the statement asserted.

No. One single hallucinated citation on a document with you as an author is not evidence of your reckless disregard for anything. These exaggerations are crazy and you would absolutely deny such accusations if you missed your co-author's AI hallucinating a citation on your manuscript too. At best it would be careless , if you really relish extrapolating from one data point and smearing people's character based on tha…

If your co-author inserted the fradulent reference, I agree that you may not have committed fraud. But your co-author did, and you didn't check their work. and knowing that you didn't check their work, you signed off on it.

You didn't pick your co-author very well, but arXiv lacks investigative powers to determine which co-author did the bad, so they all get the consequence.

Re: New arXiv policy: 1-year ban for hallucinated references

#64

Post by paulpauper »

btown wrote: post #4

> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. This is incredibly good for science. arXiv is free, but it's a privilege not a right! I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply…

My take: this seems excessive.

ArXiv doesn't even check the submission closely, so how can they know?

They say "errors, mistakes"

They use an automated system to check if the basic requirements were met, and sometimes papers are flagged for further superficial human review, but there is no way they can possibly do this at scale or check every reference. This would be like trying to do peer review, but for a preprint archive that gets easily 100x more volume than any journal.

Second, there is such a huuuuge gap between publishing on arvix and peer review. I can attest personally that it's not even close. I've gotten probably dozen rejections from peer review and no problems publishing in arxiv math. This is because peer review checks not just for if something is new or correct, but also if it's of "interest to math community," which is inherently subjective, but also makes peer review many magnitudes harder than publishing on arxiv.

Even when a well-known professor in number theory praised the paper when I got an endorsement and a second emailed me and and encouraged me to publish it, it still got rejected 3 times and still waiting.

Being required to publish in a peer reviewed journal will close off arxiv for many researchers for good. It also defeats the point of it being a pre-print.

Re: New arXiv policy: 1-year ban for hallucinated references

#65

Post by themafia »

dataflow wrote: post #18
btown wrote: post #4

> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. This is incredibly good for science. arXiv is free, but it's a privilege not a right! I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply…

> This is incredibly good for science. I disagree. It's just one darn hallucinated citation for heaven's sake, not fraud or something. It doesn't account for the substance or quality of their work at all. A one-year ban seems plenty sufficient for a minor first time mistake like this. People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's abili…

> There's no need to permanently cripple someone's ability to progress their life or contribute to humanity

I don't think you need to publish on arXive to contribute meaningfully to humanity.

> That's punitive instead of rehabilitative.

Unfortunately science is competitive. Yours is a race to the bottom where the people who can afford the most expensive models and who are least concerned with the truth can publish the most papers and benefit financially and professionally by doing so. This is not a zero sum arena, grant money and opportunities will possibly be rewarded to them, and not to another team who is producing more careful and genuine output.

Re: New arXiv policy: 1-year ban for hallucinated references

#66

Post by helterskelter »

paulpauper wrote: post #64
btown wrote: post #4

> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. This is incredibly good for science. arXiv is free, but it's a privilege not a right! I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply…

My take: this seems excessive. ArXiv doesn't even check the submission closely, so how can they know? They say "errors, mistakes" They use an automated system to check if the basic requirements were met, and sometimes papers are flagged for further superficial human review, but there is no way they can possibly do this at scale or check every reference. This would be like trying to do peer review, but for a preprint…

You could at least filter out hallucinated references which simply don't exist pretty trivially, I'd imagine.

Re: New arXiv policy: 1-year ban for hallucinated references

#68

Post by applfanboysbgon »

dataflow wrote: post #31
wrs wrote: post #22

Earlier quoted context omitted.

A "mistake" would be a typo in a real citation. A hallucinated citation is evidence of just plain laziness and negligence, which taints the entire submission.

No it is not. Seriously. All you need for this to happen is for your lab partner to ask AI to add a missing citation that they are already familiar with at the last minute before a midnight submission deadline, and for the AI to hallucinate something else, and for them to honestly miss this. It does not even imply any involvement on your part, let alone that either of you were lazy or negligent on the actual research…

Your constructed hypothetical makes it even worse. If there are 2+ people in this scenario who have good intentions, this should especially never happen. When you sign your name on a paper, you are nonetheless vouching for everything written in it, including the things you didn't personally write. You should absolutely be checking every single reference your co-author included and verifying that it says what your co-author claims it says. This is something you should have been doing completely independent of LLMs existing. This is something you're publishing publicly, something that may be associated with you and your career for the rest of your life, it is insanely negligent to not even read and verify what your co-author is adding.

Re: New arXiv policy: 1-year ban for hallucinated references

#69

Post by algorithmsRcool »

dataflow wrote: post #57
LPisGood wrote: post #53

Earlier quoted context omitted.

Fraud requires intent to deceive _or_ reckless disregard, sometimes called, “conscious indifference” for the veracity of the statement asserted.

No. One single hallucinated citation on a document with you as an author is not evidence of your reckless disregard for anything. These exaggerations are crazy and you would absolutely deny such accusations if you missed your co-author's AI hallucinating a citation on your manuscript too. At best it would be careless , if you really relish extrapolating from one data point and smearing people's character based on tha…

Allowing hallucinated content or citations into your work is an act of carelessness and disregard for the time of people that are going to read your paper and it should be policed as such.

And flatly, if a person can't be bothered to check their damn work before uploading it, why should anyone else invest their time in reading it seriously?

Re: New arXiv policy: 1-year ban for hallucinated references

#70

Post by tengwar2 »

lucb1e wrote: post #16
tengwar2 wrote: post #6

Earlier quoted context omitted.

It's not just AI, though. I did a doctorate in physics about 40 years back, and bad references were a problem back then.

In what way? Surely something like the source not quite saying what was cited, or mixing up citations, rather than inventing them outright?

In physics, references which just didn't exist. That could be that the author made it up, but often it's because they transcribed the reference from another paper without reading it - we know because a few people have deliberately introduced fake references to trace how far they would go. The reasons are not the same as for AI, but the problem they produce is the same.

References which don't accurately reflect the quoted material seem more common in other subjects.

Post reply on HN