Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New arXiv policy: 1-year ban for hallucinated references

twitter.com

Post reply on HN
Print view View original

131–140 of 242 posts

Re: New arXiv policy: 1-year ban for hallucinated references

#131

Post by fc417fc802 »

gpm wrote: post #121
fc417fc802 wrote: post #111

Earlier quoted context omitted.

I disagree with your implicit assertion regarding the common meaning of the term in this context. I believe that the term fraud as commonly used when discussing things in a scientific context has always (for at least my entire life) been taken to refer to knowingly and intentionally falsified research results (also falsified appointments, falsified affiliations, falsified authorships, etc). > deception in order to ge…

We're not "discussing things in a scientific context" here. We're in the context of a startup/programmer news aggregator discussing scientific news. We are not "a bunch of tenured career researchers" discussing amongst ourselves so the jargon appropriate for that context is not the appropriate jargon - rather we need to use the jargon that the startup/programmer news aggregator crowd would understand. That said even…

There are actually a surprising (IMO) number of career researchers on this site. Regardless, disregarding the context specific meaning will at absolute best result in a disjointed conversation where people are talking past each other. Worse, in this instance people are debating how arxiv (and other venues) ought to handle these sorts of things at which point you are well and truly into the territory where you need to get the field specific terminology right.

I concede that I was sloppy when I referred to what the researchers would be imagining. I should have phrased it as asking them if they thought that transgression X constituted fraud.

Regardless, hopefully you can see the idea that I was attempting to communicate? The burglary example isn't equivalent because while the spoons are unexpected the end result is still an event that most people would agree constituted burglary and resulted in noticeable harm to the victim.

I'm struggling to adjust your example on the fly but perhaps if it were the contents of the yard waste bin that had been pilfered? That's still technically burglary but I think most people would view it quite differently and might question the wisdom of prosecuting it.

I think the key difference here comes down to motivations as well as impact. Falsifying results (for example) is an active attempt to counterfeit the core value proposition of the endeavor and the end result of that is proportional - personal benefit directly as a result of the falsification and significant damages to anyone sufficiently bamboozled by the fiction long enough to base any decisions on it. Whereas no one using an LLM to generate just the bibliography is doing that to get ahead (at least not on its own) and any damages are limited to the reader wasting a few minutes trying to figure out the extent of the issue and who to contact about it.

Re: New arXiv policy: 1-year ban for hallucinated references

#132

Post by dataflow »

etchalon wrote: post #104
dataflow wrote: post #31

Earlier quoted context omitted.

No it is not. Seriously. All you need for this to happen is for your lab partner to ask AI to add a missing citation that they are already familiar with at the last minute before a midnight submission deadline, and for the AI to hallucinate something else, and for them to honestly miss this. It does not even imply any involvement on your part, let alone that either of you were lazy or negligent on the actual research…

You seem incredibly upset you can't get away with fraud and that people are calling it fraud.

> You seem incredibly upset you can't get away with fraud and that people are calling it fraud.

Yeah, that's exactly what I've been upset about. You really nailed it!

Re: New arXiv policy: 1-year ban for hallucinated references

#133

Post by dataflow »

dumpsterdiver wrote: post #54
dataflow wrote: post #29

Earlier quoted context omitted.

> Your being set behind is less important than the fact that your publishing is setting everyone else behind One hallucinated citation does not in any way imply anyone is being left behind. All it means nobody is checked that particular line of the manuscript after it was written. The rest of the paper could still be solid and treated accordingly. If you find evidence of the contrary, of course treat it accordingly,…

> One hallucinated citation does not in any way imply anyone is being left behind. The parent said “setting” others behind, which refers to lost time. Being “left” behind implies a degraded trajectory, which is defined not by time lost, but by the final destination. Different but related things (e.g. lost time can indeed affect your final destination, for instance, after growing old correcting a scourge of hallucinat…

That was literally just a typo, I was walking and messed up while typing. Pretend I wrote "set behind." It makes no difference to my point and I fully stand behind the comment with that correction.

If all you're genuinely worried about is the collective human time spent on tracing down one stupid hallucinated citation in a paper, may I remind you of the ludicrous amounts of time and effort readers waste trying to wade through the sea fluff, jargon, and complexity frequently added to papers in a completely deliberate fashion. If wasting even a little bit of readers' time is what you see as the crime here, you have orders of magnitude bigger fish to fry.

The fact is that, for one hallucinated citation to be the noteworthy bit that "sets others behind" in any meaningful way, the actual substance of your paper has to be utterly worthless (or worse); otherwise, you're contributing far more than you're taking away, and thus your paper is very much not setting others behind. OTOH, if your paper really is worthless or harmful enough for this part of it to be a big deal, that would be the basis for punishment, not this. A single hallucinated citation is simply not a bleep on that metaphorical radar.

Re: New arXiv policy: 1-year ban for hallucinated references

#134

Post by DANmode »

rossjudson wrote: post #124
paulpauper wrote: post #64

Earlier quoted context omitted.

My take: this seems excessive. ArXiv doesn't even check the submission closely, so how can they know? They say "errors, mistakes" They use an automated system to check if the basic requirements were met, and sometimes papers are flagged for further superficial human review, but there is no way they can possibly do this at scale or check every reference. This would be like trying to do peer review, but for a preprint…

This puts the burden to make sure it's right on the submitter, where it should be. Verification can come at any time after that; the submitter understands the consequences of hallucinated references. Verification can be crowd-sourced (and likely will be). Nothing stops someone from putting a PDF on the internet. I'm fine with ArXiv holding a high standard.

More than fine, let’s encourage it.

We deserve it, it’s one of the ways to differentiate from the Elsevier et al shitboxes!

Re: New arXiv policy: 1-year ban for hallucinated references

#135

Post by dataflow »

eqvinox wrote: post #116
ksd482 wrote: post #79

Earlier quoted context omitted.

> It is fraud. I think we are talking semantics here. While fraud does require intention to deceive, I get the sentiment that hallucinated citations shouldn't be dismissed as simply carelessness. It should be something stronger than that: gross negligence or something MUCH stronger! There should absolutely be repercussions for this. But let's not call it fraud. That word is reserved for something specific. EDIT: some…

I do believe this policy is appropriate to deal with the reckless disregard of posting hallucinated references. It's a conscious decision to not take the time to check your AI output, and instead waste a whole bunch of other people's time letting them essentially do that for you in duplicate. Feels like that should disqualify you from participation for a bit. Intent or no intent.

> Feels like that should disqualify you from participation for a bit. Intent or no intent.

Exactly! For a bit!

Yet this is not for a bit! This is a lifetime disqualification, and that's been my entire grip the whole time! Is nobody reading this?

"The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue."

Re: New arXiv policy: 1-year ban for hallucinated references

#136

Post by jruohonen »

toast0 wrote: post #33
dataflow wrote: post #18

Earlier quoted context omitted.

> This is incredibly good for science. I disagree. It's just one darn hallucinated citation for heaven's sake, not fraud or something. It doesn't account for the substance or quality of their work at all. A one-year ban seems plenty sufficient for a minor first time mistake like this. People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's abili…

> It's just one darn hallucinated citation for heaven's sake, not fraud or something. It is fraud. > It doesn't account for the substance or quality of their work at all. References are part of the work. If you're making up the references, what else are you making up? > People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's ability to progress…

> In an ideal world, one would be keeping notes on references used while doing the research that lead to writing the paper. Choosing not to do that is one poor decision.

In this book

https://news.ycombinator.com/item?id=44022957

there is this passage on p. 127:

"Any author citing another paper should be required to provide proof that they a) possess a copy of that paper, b) have read that paper, c) have read the paper carefully."

Re: New arXiv policy: 1-year ban for hallucinated references

#138

Post by small_scombrus »

nomel wrote: post #127
nradov wrote: post #118

Earlier quoted context omitted.

The frontier LLMs are getting pretty good at checking this sort of thing. You could prompt them to not only verify the references are real but that they actually state what the article claims. Some human review will still be needed but I'll bet this approach could find a lot of academic fraud.

> The frontier LLMs are getting pretty good at checking this sort of thing. No, this is career ending high stakes. it requires old school "actually check a record of reality" type methods, like a database query or http get to one of the many services that hold this info.

I think they're saying that frontier LLMs may be usable to spot citations that are correct by shape (a real citation) but incorrect by usage (unrelated to the text)

I kind of hate the idea, but you probably could do a lazy LLM check of every paper and every citation and have it flag possible wrong (second sense) citations for human review

But you'd need a LOT of tokens and a LOT of human-hours

Re: New arXiv policy: 1-year ban for hallucinated references

#139

Post by small_scombrus »

ktallett wrote: post #73
dataflow wrote: post #57

Earlier quoted context omitted.

No. One single hallucinated citation on a document with you as an author is not evidence of your reckless disregard for anything. These exaggerations are crazy and you would absolutely deny such accusations if you missed your co-author's AI hallucinating a citation on your manuscript too. At best it would be careless , if you really relish extrapolating from one data point and smearing people's character based on tha…

How are you suggesting the fake citation came about? Why are you writing papers and not having actually read the source you took the material from?

> Why are you writing papers and not having actually read the source you took the material from?

They're explicitly not writing papers. The fake citations are created and inserted by the LLM

Post reply on HN