Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New arXiv policy: 1-year ban for hallucinated references

twitter.com

Post reply on HN
Print view View original

211–220 of 242 posts

Re: New arXiv policy: 1-year ban for hallucinated references

#211

Post by clearstack »

kevincox wrote: post #195
clearstack wrote: post #192

the hallucination problem is especially bad in financial AI. one made-up revenue number and someone makes a bad investment. the actual 10-K is the only source of truth for company data

That sounds like way less of a problem than science. If you take bad sources you hurt yourself financially. For science (especially anything medicine related) you can hurt many others physically.

fair at the individual level. but now these tools are used in quant models and at retail scale — one systematically hallucinated metric can affect a lot of real portfolios

Re: New arXiv policy: 1-year ban for hallucinated references

#213

Post by lbrito »

vwkd wrote: post #207

Previously: „Generate a bibliography.“ Now: „Generate a bibliography. Make no mistakes.“

As another comment said (not sure if sarcastically or not - impossible to tell): "just spin up another agent(s) to check the work of the first!"

Re: New arXiv policy: 1-year ban for hallucinated references

#214

Post by godelski »

withinboredom wrote: post #154
godelski wrote: post #100

Earlier quoted context omitted.

> In an ideal world, one would be keeping notes on references used In a far less than ideal world authors are referencing papers they've at least read the title and abstract of. In an ideal world, authors would be only referencing works they have read in their entirety. I don't think we need to live in the ideal world[0], but let's also not pretend the ideal world is even remotely out of reach. Let's also be honest t…

Even as a human, you can still fuck up references. I submitted a paper with a reference author as Elisio because I couldn’t read my own handwriting. After submitting, I double checked all the references through an LLM. It pointed out that their name was actually Enrique. Yes, you should probably double check your references before submitting, not after. Point is, I didn’t even trust the LLM at first. But after verify…

Given their examples and examples I've seem Thomas talk about in the past, I doubt a typo like that would be grounds for the ban.

Perhaps the issue is that people aren't logged in or using xcancel so missing part of the tweet thread. Here's an important line

> If a submission contains incontrovertible evidence that the authors did not check the results of LLM generation, this means we can't trust anything in the paper.

Followed by

> Examples of incontrovertible evidence: hallucinated references, meta-comments from the LLM ("here is a 200 word summary; would you like me to make any changes?"; "the data in this table is illustrative, fill it in with the real numbers from your experiments")

I wouldn't look at your case and read that as "incontrovertible evidence". They are looking for the absolutely brain dead, no one at the wheel, type of errors. They're looking for things like your paper saying "As an AI language model". Which, there will be real papers with that exact phrase, but it should get flagged, not auto banned

Re: New arXiv policy: 1-year ban for hallucinated references

#215

Post by gpm »

fc417fc802 wrote: post #131
gpm wrote: post #121

Earlier quoted context omitted.

We're not "discussing things in a scientific context" here. We're in the context of a startup/programmer news aggregator discussing scientific news. We are not "a bunch of tenured career researchers" discussing amongst ourselves so the jargon appropriate for that context is not the appropriate jargon - rather we need to use the jargon that the startup/programmer news aggregator crowd would understand. That said even…

There are actually a surprising (IMO) number of career researchers on this site. Regardless, disregarding the context specific meaning will at absolute best result in a disjointed conversation where people are talking past each other. Worse, in this instance people are debating how arxiv (and other venues) ought to handle these sorts of things at which point you are well and truly into the territory where you need to…

> I should have phrased it as asking them if they thought that transgression X constituted fraud.

I'm not sure you're right that they wouldn't think it's fraud at first glance, but I also think it's the wrong test. Words have meanings that result in an actual truth to the matter that is sometimes non-obvious.

A theorem isn't false just because it intuitively seems false until you spend decades trying to prove it and finally find the proof. Requiring people be cleanshaven isn't obviously discriminatory, until someone points out the existence and prevalence of pseudofolliculitis barbae and perhaps that an action being discriminatory doesn't actually require intent.

So I'd propose the correct test (and again, this would only be for a discussion in an academic context, which this is not) looks something like giving them access to the common dictionary definitions, and legal definitions, of fraud, and then asking them to define it. Then taking their definition and applying it to the facts at hand.

I don't think your average academic thinks academia has a special meaning of the word. I think they would simply acknowledge the dictionary definition as the correct one, and as a result I think this is plainly fraud even as the word is used in an academic context.

---

I don't think anyone defines fraud as only deception relating to the core value proposition of the instrument of the deception.

Re: New arXiv policy: 1-year ban for hallucinated references

#216

Post by toast0 »

DiogenesKynikos wrote: post #128
toast0 wrote: post #63

Earlier quoted context omitted.

If your co-author inserted the fradulent reference, I agree that you may not have committed fraud. But your co-author did, and you didn't check their work. and knowing that you didn't check their work, you signed off on it. You didn't pick your co-author very well, but arXiv lacks investigative powers to determine which co-author did the bad, so they all get the consequence.

Do you think every co-author on a 100-author paper checks every citation? It's like saying that every member of a large software team personally reviews every line of code. It's just completely divorced from reality.

How many papers have 100 authors?

Again, I'm not in academia, but most of the papers I see have two to five authors; maybe I've seen one or two with ten authors.

Regardless, if I'm signing my name on something, I check it out.

A reference that doesn't exist is like a source release that doesn't compile. Any one of the 100 person team could have figured it out but no one did. In a 100 person team you get diffuse responsibility dynamics where checking citations is not assigned to specific people so no one does it. Or perhaps it's assigned to a single person who was also in charge of writing the citations and falsified them rather than doing the work.

Re: New arXiv policy: 1-year ban for hallucinated references

#218

Post by avanwyk »

dataflow wrote: post #18
btown wrote: post #4

> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. This is incredibly good for science. arXiv is free, but it's a privilege not a right! I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply…

> This is incredibly good for science. I disagree. It's just one darn hallucinated citation for heaven's sake, not fraud or something. It doesn't account for the substance or quality of their work at all. A one-year ban seems plenty sufficient for a minor first time mistake like this. People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's abili…

We got pretty far without a single hallucinated citation (because they simply didn't exist). It's really simple: check your references, or don't bother of publishing. It's not a high hurdle.

Re: New arXiv policy: 1-year ban for hallucinated references

#219

Post by jkman »

____mr____ wrote: post #194
btown wrote: post #4

> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. This is incredibly good for science. arXiv is free, but it's a privilege not a right! I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply…

> subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. Isn't the whole point of arxiv to be somewhere you post your low-reviewed/unpublished/prepublished papers? doesnt this somewhat defeat the point and basically means a lifetime ban?

Eh guess so, I think it's appropriate

Re: New arXiv policy: 1-year ban for hallucinated references

#220

Post by T-A »

btown wrote: post #4

> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. This is incredibly good for science. arXiv is free, but it's a privilege not a right! I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply…

>> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue.

Catch-22 [1]:

You will need to provide an arXiv article id. This is necessary for your paper to be processed correctly, the submitted version must be the same as the arXiv version.

[1] https://jcap.sissa.it/jcap/help/helpLoader.jsp?pgType=author

Post reply on HN