Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New arXiv policy: 1-year ban for hallucinated references

twitter.com

Post reply on HN
Print view View original

191–200 of 242 posts

Re: New arXiv policy: 1-year ban for hallucinated references

#191

Post by mpalmer »

vrighter wrote: post #168
nradov wrote: post #118

Earlier quoted context omitted.

The frontier LLMs are getting pretty good at checking this sort of thing. You could prompt them to not only verify the references are real but that they actually state what the article claims. Some human review will still be needed but I'll bet this approach could find a lot of academic fraud.

why is the standard response to "this tech isn't reliable enough for this" to run its output through the same unreliable tech? The device-fixer started breaking devices instead of fixing them. Tell it to fix itself!

Yeah...

The amount of people who confidently tell on themselves in these discussions continues to bum me out.

Re: New arXiv policy: 1-year ban for hallucinated references

#193

Post by skydhash »

fc417fc802 wrote: post #122
andrepd wrote: post #48

Earlier quoted context omitted.

If you are citing a work you paste a citation to that work. If you are bullshitting you ask an AI to come up with a citation. Jesus, there is zero reason to ever "generate a citation" if you are not, in fact, commiting fraud.

That's like saying that there's zero reason to ever ask an LLM to do basic math for you. Sure you probably shouldn't do that but sometimes it's convenient and so people will inevitably do exactly that regardless of the somewhat frequent wrong answers that are guaranteed to ensue.

Fraud are always about the convenience of the perpetrator.

Re: New arXiv policy: 1-year ban for hallucinated references

#194

Post by ____mr____ »

btown wrote: post #4

> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. This is incredibly good for science. arXiv is free, but it's a privilege not a right! I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply…

> subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue. Isn't the whole point of arxiv to be somewhere you post your low-reviewed/unpublished/prepublished papers? doesnt this somewhat defeat the point and basically means a lifetime ban?

Re: New arXiv policy: 1-year ban for hallucinated references

#195

Post by kevincox »

clearstack wrote: post #192

the hallucination problem is especially bad in financial AI. one made-up revenue number and someone makes a bad investment. the actual 10-K is the only source of truth for company data

That sounds like way less of a problem than science. If you take bad sources you hurt yourself financially. For science (especially anything medicine related) you can hurt many others physically.

Re: New arXiv policy: 1-year ban for hallucinated references

#196

Post by skydhash »

soraminazuki wrote: post #177

It's not unexpected, but still sad to see so many comments opposing even the smallest step against low-effort fraud in academic publications. Is this what hacker culture has been reduced to in the age of the slop era? Open hostility against science and engineering?

Something something about not understanding the problem when your salary depends on it…

Re: New arXiv policy: 1-year ban for hallucinated references

#197

Post by CaptainNegative »

vrighter wrote: post #168
nradov wrote: post #118

Earlier quoted context omitted.

The frontier LLMs are getting pretty good at checking this sort of thing. You could prompt them to not only verify the references are real but that they actually state what the article claims. Some human review will still be needed but I'll bet this approach could find a lot of academic fraud.

why is the standard response to "this tech isn't reliable enough for this" to run its output through the same unreliable tech? The device-fixer started breaking devices instead of fixing them. Tell it to fix itself!

why is the standard response when someone comes down with a serious illness to bring them into a facility where serious illnesses spread readily?

sometimes the presently available solutions are subpar. people go with what's available. it's not ideal, but it is practical.

Re: New arXiv policy: 1-year ban for hallucinated references

#198

Post by cyclecycle »

gizajob wrote: post #173
cyclecycle wrote: post #153

This has become such a problem in scholarly publishing that we have a business that provides citation checking https://groundedai.company/ that we've been buidling for a couple of years now

What’s the hallucination rate of your AI?

So far we basically just provide a very rule-based approach and try not use LLMs as much as possible. So we extract and parse the citations using various ML and rule-based approaches, and carry out a bunch of predetermined queries and do various fuzzy matching approaches on the metadata components, and have a bunch of rules around risk levels of things we should have found/matched based on what type of source it is, which venue we should have found it in, etc.

So there are absolutely a bunch of tasks that could be evaled/benchmarked, but "hallucination rate" isn't particularly applicable/interesting as a metric of how good the tool is

that said, we do use various LLMs (mostly local, fine-tuned, small, for things like NER/parsing/metadata comparison, etc.). and they can and do hallucinate, but we have very hard constraints on the validation, so any extraction results that don't match 1:1 back to the input text are discarded for example. so again, rather than hallucination risk we prefer hard constraints

Re: New arXiv policy: 1-year ban for hallucinated references

#199

Post by user34283 »

noobermin wrote: post #60

Seeing the usual LLM hypers angry replying to this on twitter is such a tell. Just like the comments on the LLM poisoning articles, some people just can't accept that some people don't like LLMs and get upset when you put any amount of hindrance to their rapid acceptance.

While this arXiv policy seems reasonable enough, I don't care for the kind of drivel some post on HN because they don't like LLMs.

I'm here because I enjoy building things. And today this mostly happens with AI. I could do without the often thoughtless comments and conspiracy theories about "LLM hypers" posted by people who don't like LLMs.

Re: New arXiv policy: 1-year ban for hallucinated references

#200

Post by 8organicbits »

mpalmer wrote: post #190
small_scombrus wrote: post #138

Earlier quoted context omitted.

I think they're saying that frontier LLMs may be usable to spot citations that are correct by shape (a real citation) but incorrect by usage (unrelated to the text) I kind of hate the idea, but you probably could do a lazy LLM check of every paper and every citation and have it flag possible wrong (second sense) citations for human review But you'd need a LOT of tokens and a LOT of human-hours

> have it flag possible wrong (second sense) citations for human review And then what, we're done? How have we avoided the need for the same exhaustive human review? It only saves human review time if you trust the LLM not to miss things.

If the goal is to review every citation fully with 100% accuracy, then, sure, exhaustive human review is needed. But I suspect human review of a random sample would add value, catching some fraud, missing others, but having zero false positives (or as close to zero as human review can get).

An LLM could replace the random sampling. It doesn't need to be particularly good for the approach to provide value. I would worry about LLM bias though.

Another thing to consider is that readers can detect fake citations after publication, report to arXiv, and the author gets banned.

Post reply on HN