Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New arXiv policy: 1-year ban for hallucinated references

twitter.com

Post reply on HN
Print view View original

111–120 of 242 posts

Re: New arXiv policy: 1-year ban for hallucinated references

#111

Post by fc417fc802 »

gpm wrote: post #109
fc417fc802 wrote: post #103

Earlier quoted context omitted.

If I violate the letter of the ToS when clicking submit you can correctly argue that I have technically committed fraud! Yet that is almost never what anyone actually means when having discussions like this one. Fraud in a scientific context generally refers to fabricated research results. At least personally I agree with GP that hallucinated citations are generally something akin to laziness thus not fraud but rathe…

Fraud in the scientific world has generally taken the form of fabricated results, but I don't agree that the word has transitioned away from the common and legal meaning of deception in order to get a benefit. Even if it had though, I'd be perfectly comfortable calling this fraud in this discussion based on the common meaning of the word. Just because we're talking about a scientific context does not mean we need to…

I disagree with your implicit assertion regarding the common meaning of the term in this context. I believe that the term fraud as commonly used when discussing things in a scientific context has always (for at least my entire life) been taken to refer to knowingly and intentionally falsified research results (also falsified appointments, falsified affiliations, falsified authorships, etc).

> deception in order to get a benefit.

The point being that reckless or negligent conduct is not commonly taken to constitute deception. There's a reason we have different terms for these things.

Sure, you can say "well he exhibited reckless disregard for his professional duties when he opted not to bother reading the citation section that the LLM shat out, and reckless disregard is sufficient to meet the legal bar for fraud, and also the ToS specifically says that you certify that you validated all references manually so bam! two counts of fraud legally speaking" and you wouldn't be wrong but the distinction between "legally fraud" and "fraud as is commonly meant when talking about scientific papers" is essential to effective conversation in this particular instance.

> Just because we're talking about a scientific context does not mean we need to use the scientific-jargon versions of words

The context is essential because (obviously) it affects how people interpret the meaning of your words. A fraudulent submission to a scientific journal has a specific and well understood meaning in common usage.

If you still disagree with me imagine polling a bunch of tenured career researchers about what they would think if they read the statement "X caught submitting a fraudulent paper to journal Y". I can just about guarantee you that none of them are imagining hallucinated citations.

Re: New arXiv policy: 1-year ban for hallucinated references

#112

Post by eqvinox »

paulpauper wrote: post #64
btown wrote: post #4

> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. This is incredibly good for science. arXiv is free, but it's a privilege not a right! I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply…

My take: this seems excessive. ArXiv doesn't even check the submission closely, so how can they know? They say "errors, mistakes" They use an automated system to check if the basic requirements were met, and sometimes papers are flagged for further superficial human review, but there is no way they can possibly do this at scale or check every reference. This would be like trying to do peer review, but for a preprint…

You don't need to be actively enforcing a rule 100% on everyone. Speed cameras don't cover every stretch of road either.

It's enough for them to place this policy and enforce it when they become aware of violations. Someone reading the slopped paper (or, here, trying to follow a reference) will notice sooner or later.

> Being required to publish in a peer reviewed journal will close off arxiv for many researchers for good. It also defeats the point of it being a pre-print.

You sound like it's impossible for researchers to write papers without slopped references, and inevitable to get hit by this policy.

Re: New arXiv policy: 1-year ban for hallucinated references

#114

Post by pinkmuffinere »

kingstnap wrote: post #14
pinkmuffinere wrote: post #11

Earlier quoted context omitted.

> think proof of Riemann Hypothesis autonomously generated (end to end) formally proven - is it allowed or not? Sorry to be rude, but this seems like a dumb question. I want science to progress. A primary purpose of these journals is to progress science. A full proof of the Riemann Hypothesis progresses science. I don't care how it was produced, if Hitler is coauthor, etc, I just care that it is correct. Whether the…

Terence Tao had a nice talk from the Future of Mathematics conference posted yesterday [0] that shapes a lot of my own feelings on this matter. The short of it is he argues how first to correctness shouldn't be the only goal / isn't a great optimisation incentive. Presentation and digestibility of correct results is a missing 1/3 when you've finished generation and verification. I completely agree with him. You don't…

Ya, I think this totally makes sense. Just to be clear though, I don’t think we’re actually disagreeing. A proof of the Riemann hypothesis that’s obtuse and basically unreadable is a great step on the path to a proof that is enlightening and clear. If ai provides correct-but-annoying results, I’m confident humans can still learn benefit from that marginal result.

Re: New arXiv policy: 1-year ban for hallucinated references

#115

Post by casey2 »

eqvinox wrote: post #112
paulpauper wrote: post #64

Earlier quoted context omitted.

My take: this seems excessive. ArXiv doesn't even check the submission closely, so how can they know? They say "errors, mistakes" They use an automated system to check if the basic requirements were met, and sometimes papers are flagged for further superficial human review, but there is no way they can possibly do this at scale or check every reference. This would be like trying to do peer review, but for a preprint…

You don't need to be actively enforcing a rule 100% on everyone. Speed cameras don't cover every stretch of road either. It's enough for them to place this policy and enforce it when they become aware of violations. Someone reading the slopped paper (or, here, trying to follow a reference) will notice sooner or later. > Being required to publish in a peer reviewed journal will close off arxiv for many researchers for…

Even acts that would be criminal in the US occur less in China due to properly enforced fines. Nobody does things assuming they will get caught unless there is a high likelihood of getting caught.

Research and practice has shown that the strongest deterrent is certainty.

Re: New arXiv policy: 1-year ban for hallucinated references

#116

Post by eqvinox »

ksd482 wrote: post #79
toast0 wrote: post #33

Earlier quoted context omitted.

> It's just one darn hallucinated citation for heaven's sake, not fraud or something. It is fraud. > It doesn't account for the substance or quality of their work at all. References are part of the work. If you're making up the references, what else are you making up? > People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's ability to progress…

> It is fraud. I think we are talking semantics here. While fraud does require intention to deceive, I get the sentiment that hallucinated citations shouldn't be dismissed as simply carelessness. It should be something stronger than that: gross negligence or something MUCH stronger! There should absolutely be repercussions for this. But let's not call it fraud. That word is reserved for something specific. EDIT: some…

I do believe this policy is appropriate to deal with the reckless disregard of posting hallucinated references.

It's a conscious decision to not take the time to check your AI output, and instead waste a whole bunch of other people's time letting them essentially do that for you in duplicate.

Feels like that should disqualify you from participation for a bit. Intent or no intent.

Re: New arXiv policy: 1-year ban for hallucinated references

#117

Post by fc417fc802 »

NiloCK wrote: post #52
dataflow wrote: post #41

Earlier quoted context omitted.

> It is fraud. No, it is emphatically not. Fraud requires intent to deceive. > A one year ban is not permanent. ...what text are you reading? Nobody was calling the one-year ban permanent, or even against it. I was literally in favor of it in my comment. I explicitly said it is already plenty sufficient. What I said is there's no need to go beyond that. My entire gripe was that they very much are going beyond that wi…

> No, it is emphatically not. D Fraud requires intent to deceive. I'm about as pro AI-as-a-research--and-writing-assistant and anti AI-witchhunt as they come, but I simply cannot parse what I've quoted here. Posting slop to arxiv is blatant deception. Posting an article is an attestation that the article is a genuine engagement with the literature . If you're posting things to arxiv that are not sincere engagements w…

You are equating cutting corners (ie laziness) with intentional deception and not being genuine. That doesn't seem accurate to me. In most contexts I think cutting corners would be taken to be some form of negligence or recklessness.

Regardless of terminology, I agree that it's certainly punishable and certainly a serious problem.

Re: New arXiv policy: 1-year ban for hallucinated references

#118

Post by nradov »

nomel wrote: post #96
btown wrote: post #4

> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. This is incredibly good for science. arXiv is free, but it's a privilege not a right! I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply…

I bet, since this has been posted, someone here has already vibe coded a reference checker that they plan to put behind a subscription. This is good for reference checking, but I doubt this will do much for the most likely shoddy science that accompanies hallucinated references.

The frontier LLMs are getting pretty good at checking this sort of thing. You could prompt them to not only verify the references are real but that they actually state what the article claims. Some human review will still be needed but I'll bet this approach could find a lot of academic fraud.

Re: New arXiv policy: 1-year ban for hallucinated references

#119

Post by [deleted] »

fc417fc802 wrote: post #117
NiloCK wrote: post #52

Earlier quoted context omitted.

> No, it is emphatically not. D Fraud requires intent to deceive. I'm about as pro AI-as-a-research--and-writing-assistant and anti AI-witchhunt as they come, but I simply cannot parse what I've quoted here. Posting slop to arxiv is blatant deception. Posting an article is an attestation that the article is a genuine engagement with the literature . If you're posting things to arxiv that are not sincere engagements w…

You are equating cutting corners (ie laziness) with intentional deception and not being genuine. That doesn't seem accurate to me. In most contexts I think cutting corners would be taken to be some form of negligence or recklessness. Regardless of terminology, I agree that it's certainly punishable and certainly a serious problem.

[deleted]

Re: New arXiv policy: 1-year ban for hallucinated references

#120

Post by fc417fc802 »

wrs wrote: post #86
dataflow wrote: post #31

Earlier quoted context omitted.

No it is not. Seriously. All you need for this to happen is for your lab partner to ask AI to add a missing citation that they are already familiar with at the last minute before a midnight submission deadline, and for the AI to hallucinate something else, and for them to honestly miss this. It does not even imply any involvement on your part, let alone that either of you were lazy or negligent on the actual research…

You’re confusing the issue here by saying it’s not your fault, it’s your lab partner’s. We’re talking about why your lab partner did something wrong. You can assign blame for the wrong thing separately. The citation is part of the substance of the paper . If you YOLOed in a citation without checking it, seems justified to suspect that you may have YOLOed in some data, or some analysis, or maybe even the conclusion.

Being suspicious would be reasonable and I think a penalty could perhaps be appropriate but the person you're replying to is objecting that the stated hypothetical would not rise to the bar of negligence.

Do bear in mind the degree of the described scenario. There's quite a difference between having an LLM shit out your entire citation section (and possibly the rest of the paper as well) versus asking the tool to make a targeted edit and overlooking a small piece of nonsense that results.

Post reply on HN