Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New arXiv policy: 1-year ban for hallucinated references

twitter.com

Post reply on HN
Print view View original

71–80 of 242 posts

Re: New arXiv policy: 1-year ban for hallucinated references

#71

Post by protocolture »

NiloCK wrote: post #52
dataflow wrote: post #41

Earlier quoted context omitted.

> It is fraud. No, it is emphatically not. Fraud requires intent to deceive. > A one year ban is not permanent. ...what text are you reading? Nobody was calling the one-year ban permanent, or even against it. I was literally in favor of it in my comment. I explicitly said it is already plenty sufficient. What I said is there's no need to go beyond that. My entire gripe was that they very much are going beyond that wi…

> No, it is emphatically not. D Fraud requires intent to deceive. I'm about as pro AI-as-a-research--and-writing-assistant and anti AI-witchhunt as they come, but I simply cannot parse what I've quoted here. Posting slop to arxiv is blatant deception. Posting an article is an attestation that the article is a genuine engagement with the literature . If you're posting things to arxiv that are not sincere engagements w…

>I'm about as pro AI-as-a-research--and-writing-assistant and anti AI-witchhunt as they come, but I simply cannot parse what I've quoted here.

Ditto. And its only 1 year. Like its about the most reasonable thing they could have done.

Re: New arXiv policy: 1-year ban for hallucinated references

#72

Post by vhantz »

dataflow wrote: post #18
btown wrote: post #4

> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. This is incredibly good for science. arXiv is free, but it's a privilege not a right! I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply…

> This is incredibly good for science. I disagree. It's just one darn hallucinated citation for heaven's sake, not fraud or something. It doesn't account for the substance or quality of their work at all. A one-year ban seems plenty sufficient for a minor first time mistake like this. People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's abili…

What's the difference between a "hallucinated" citation and consciously inserting reference to a non-existent paper and hopping it goes unnoticed? How do we determine which one was done consciously and which was "a minor first time mistake"?

Your standards are lower than what they would accept at my high-school. Seriously.

And generally, if you are generating papers with LLMs, let other LLMs read them. Why would we waste human hours considering something that was generated? At this point publish your prompt because that's the actual work you're doing.

Re: New arXiv policy: 1-year ban for hallucinated references

#73

Post by ktallett »

dataflow wrote: post #57
LPisGood wrote: post #53

Earlier quoted context omitted.

Fraud requires intent to deceive _or_ reckless disregard, sometimes called, “conscious indifference” for the veracity of the statement asserted.

No. One single hallucinated citation on a document with you as an author is not evidence of your reckless disregard for anything. These exaggerations are crazy and you would absolutely deny such accusations if you missed your co-author's AI hallucinating a citation on your manuscript too. At best it would be careless , if you really relish extrapolating from one data point and smearing people's character based on tha…

How are you suggesting the fake citation came about? Why are you writing papers and not having actually read the source you took the material from?

Re: New arXiv policy: 1-year ban for hallucinated references

#74

Post by tengwar2 »

wrs wrote: post #25
tengwar2 wrote: post #6

Earlier quoted context omitted.

It's not just AI, though. I did a doctorate in physics about 40 years back, and bad references were a problem back then.

"Bad", like, you literally just made them up? I hope that would have been a problem.

Well, I certainly didn't make them up. But it was common to follow a reference and find that there was no paper on the other end.

Re: New arXiv policy: 1-year ban for hallucinated references

#75

Post by ajkjk »

dataflow wrote: post #24
ajkjk wrote: post #21

Earlier quoted context omitted.

It's not the kind of mistake that is possible unless you're engaging in fraud anyway.

> It's not the kind of mistake that is possible unless you're engaging in fraud anyway. Seriously? You can't fathom an honest researcher asking for AI to find a citation they know exists, and the AI inserting or modifying a citation incorrectly without them realizing? If you find evidence of fraud by all means lay down the hammer. Using a single hallucinated citation like it's some kind of ironclad proxy just because…

Yes, having AI write something and not checking it yourself is sure to lead to hallucination, hence, it is a fraudulent way to write.

Re: New arXiv policy: 1-year ban for hallucinated references

#76

Post by no-name-here »

galaxyLogic wrote: post #58
andrepd wrote: post #48

Earlier quoted context omitted.

If you are citing a work you paste a citation to that work. If you are bullshitting you ask an AI to come up with a citation. Jesus, there is zero reason to ever "generate a citation" if you are not, in fact, commiting fraud.

I much agree. But I wonder shouldn't the citations all be hyperlinks and thus easy to verify?

How specific are the citations? If it's “Sentence 4 on page 97 supports” or “Paper says ‘___’” then I imagine it would be fairly easy. If it's “(__ page long) paper supports x”, then very difficult?

Re: New arXiv policy: 1-year ban for hallucinated references

#77

Post by protocolture »

dataflow wrote: post #24
ajkjk wrote: post #21

Earlier quoted context omitted.

It's not the kind of mistake that is possible unless you're engaging in fraud anyway.

> It's not the kind of mistake that is possible unless you're engaging in fraud anyway. Seriously? You can't fathom an honest researcher asking for AI to find a citation they know exists, and the AI inserting or modifying a citation incorrectly without them realizing? If you find evidence of fraud by all means lay down the hammer. Using a single hallucinated citation like it's some kind of ironclad proxy just because…

>Seriously? You can't fathom an honest researcher asking for AI to find a citation they know exists

Assumptions:

1. The entire document is loaded into an AI editor

2. The researcher is asking an AI editor to work on his references

3. The researcher has not checked his own references.

This could be avoided at 1, 2 or 3. But even just 1 implies that the researcher knows that they have a hot potato and might critically fuck up and lose all credibility. Being in that scenario and committing to 2 and 3 is at least extreme negligence.

Re: New arXiv policy: 1-year ban for hallucinated references

#78

Post by paulpauper »

helterskelter wrote: post #66
paulpauper wrote: post #64

Earlier quoted context omitted.

My take: this seems excessive. ArXiv doesn't even check the submission closely, so how can they know? They say "errors, mistakes" They use an automated system to check if the basic requirements were met, and sometimes papers are flagged for further superficial human review, but there is no way they can possibly do this at scale or check every reference. This would be like trying to do peer review, but for a preprint…

You could at least filter out hallucinated references which simply don't exist pretty trivially, I'd imagine.

It's more than that. if there are mistakes, then you can also be flagged.

read the whole tweet:

If generative AI tools generate inappropriate language, plagiarized content, biased content, errors, mistakes, incorrect references, or misleading content, and that output is included in scientific works, it is the responsibility of the author(s).

Re: New arXiv policy: 1-year ban for hallucinated references

#79

Post by ksd482 »

toast0 wrote: post #33
dataflow wrote: post #18

Earlier quoted context omitted.

> This is incredibly good for science. I disagree. It's just one darn hallucinated citation for heaven's sake, not fraud or something. It doesn't account for the substance or quality of their work at all. A one-year ban seems plenty sufficient for a minor first time mistake like this. People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's abili…

> It's just one darn hallucinated citation for heaven's sake, not fraud or something. It is fraud. > It doesn't account for the substance or quality of their work at all. References are part of the work. If you're making up the references, what else are you making up? > People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's ability to progress…

> It is fraud.

I think we are talking semantics here.

While fraud does require intention to deceive, I get the sentiment that hallucinated citations shouldn't be dismissed as simply carelessness. It should be something stronger than that: gross negligence or something MUCH stronger! There should absolutely be repercussions for this.

But let's not call it fraud. That word is reserved for something specific.

EDIT: someone else said "reckless disregard" equals intent or something to that effect. So I looked it up.

It appears so that is the case. "Reckless Disregard Equals Intent" in legal language.

But I am not sure if this particular clause should apply here. Perhaps it depends on what kind of research is being published? For e.g., if it is related to medical science and has a real consequence on people's health, we can then apply this?

Re: New arXiv policy: 1-year ban for hallucinated references

#80

Post by noobermin »

tengwar2 wrote: post #6
bigfishrunning wrote: post #3

Good; academic literature is in crisis because of all of the slop. Forcing some consequences on easily-detectable hallucinations can only be a good thing

It's not just AI, though. I did a doctorate in physics about 40 years back, and bad references were a problem back then.

Which is why the angry replies on Twitter from AI hype accounts is so funny. You should get penalised for fake references and profanity in your submissions, even if you wrote your slop longhand. I don't know why anyone would have an issue with this policy.
Post reply on HN