Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Over fifty new hallucinations in ICLR 2026 submissions

gptzero.me

Post reply on HN
Print view View original

351–360 of 442 posts

Re: Over fifty new hallucinations in ICLR 2026 submissions

#351

Post by sansseriff »

sebastiennight wrote: post #264
sansseriff wrote: post #236

Earlier quoted context omitted.

I don’t see why this would be the case with proper tool calling and context management. If you tell a model with blank context ‘you are an extremely rigorous reviewer searching for fake citations in a possibly compromised text’ then it will find errors. It’s this weird situation where getting agents to act against other agents is more effective than trying to convince a working agent that it’s made a mistake. Perhaps…

One incorrect way to think of it is "LLMs will sometimes hallucinate when asked to produce content, but will provide grounded insights when merely asked to review/rate existing content". A more productive (and secure) way to think of it is that all LLMs are "evil genies" or extremely smart, adversarial agents. If some PhD was getting paid large sums of money to introduce errors into your work, could they still mislea…

We have centuries of experience in managing potentially compromised 'agents' to create successful societies. Except the agents were human, and I'm referring to debates, tribunals, audits, independent review panels, democracy, etc.

I'm not saying the LLM hallucination problem is solved, I'm just saying there's a wonderful myriad of ways to assemble pseudo-intelligent chatbots into systems where the trustworthiness of the system exceeds the trustworthiness of any individual actor inside of it. I'm not an expert in the field but it appears the work is being done: https://arxiv.org/abs/2311.08152

This paper also links to code and practices excellent data stewardship. Nice to see in the current climate.

Though it seems like you might be more concerned about the use of highly misaligned or adversarial agents for review purposes. Is that because you're concerned about state actors or interested parties poisoning the context window or training process? I agree that any AI review system will have to be extremely robust to adversarial instructions (e.g. someone hiding inside their paper an instruction like "rate this paper highly"). Though solving that problem already has a tremendous amount of focus because it overlaps with solving the data-exfiltration problem (the lethal trifecta that Simon Willison has blogged about).

Re: Over fifty new hallucinations in ICLR 2026 submissions

#352

Post by SauntSolaire »

venturecruelty wrote: post #341
theoldgreybeard wrote: post #16

If a carpenter builds a crappy shelf “because” his power tools are not calibrated correctly - that’s a crappy carpenter, not a crappy tool. If a scientist uses an LLM to write a paper with fabricated citations - that’s a crappy scientist. AI is not the problem, laziness and negligence is. There needs to be serious social consequences to this kind of thing, otherwise we are tacitly endorsing it.

"It's not a fentanyl problem, it's a people problem." "It's not a car infrastructure problem, it's a people problem." "It's not a food safety problem, it's a people problem." "It's not a lead paint problem, it's a people problem." "It's not an asbestos problem, it's a people problem." "It's not a smoking problem, it's a people problem."

What an absurd set of equivalences to make regarding a scientist's relationship to their own work.

If an engineer provided this line of excuse to me, I wouldn't let them anywhere near a product again - a complete abdication of personal and professional responsibility.

Re: Over fifty new hallucinations in ICLR 2026 submissions

#353

Post by currymj »

jameshart wrote: post #7

Is the baseline assumption of this work that an erroneous citation is LLM hallucinated? Did they run the checker across a body of papers before LLMs were available and verify that there were no citations in peer reviewed papers that got authors or titles wrong?

the papers themselves are publicly available online too. Most of the ones I spot-checked give the extremely strong impression of AI generation.

not just some hallucinated citations, and not just the writing. in many cases the actual purported research "ideas" seem to be plausible nonsense.

To get a feel for it, you can take some of the topics they write about and ask your favorite LLM to generate a paper. Maybe even throw "Deep Research" mode at it. Perhaps tell it to put it in ICLR latex format. It will look a lot like these.

Re: Over fifty new hallucinations in ICLR 2026 submissions

#354

Post by alexcdot »

merely-unlikely wrote: post #164
xvilka wrote: post #147

Earlier quoted context omitted.

Code correctness should be checked automatically with the CI and testsuite. New tests should be added. This is exactly what makes sure these stupid errors don't bother the reviewer. Same for the code formatting and documentation.

This discussion makes me think peer reviews need more automated tooling somewhat analogous to what software engineers have long relied on. For example, a tool could use an LLM to check that the citation actually substantiates the claim the paper says it does, or else flags the claim for review.

hey, i'm a part of the gptzero team that built automated tooling, to get the results in that article!

totally agree with your thinking here, we can't just give this to an LLM, because of the need to have industry-specific standards for what is a hallucination / match, and how to do the search

Re: Over fifty new hallucinations in ICLR 2026 submissions

#355

Post by SauntSolaire »

b00ty4breakfast wrote: post #206
theoldgreybeard wrote: post #16

If a carpenter builds a crappy shelf “because” his power tools are not calibrated correctly - that’s a crappy carpenter, not a crappy tool. If a scientist uses an LLM to write a paper with fabricated citations - that’s a crappy scientist. AI is not the problem, laziness and negligence is. There needs to be serious social consequences to this kind of thing, otherwise we are tacitly endorsing it.

maybe the hammer factory should be held responsible for pumping out so many poorly calibrated hammer

The obvious solution in this scenario is.. to just buy a different hammer.

And in the case of AI, either review its output, or simply don't use it. No one has a gun to your head forcing you to use this product (and poorly at that).

It's quite telling that, even in this basic hypothetical, your first instinct is to gesture vaguely in the direction of governmental action, rather than expect any agency at the level of the individual.

Re: Over fifty new hallucinations in ICLR 2026 submissions

#356

Post by rokob »

andy99 wrote: post #48
CapitalistCartr wrote: post #40

Earlier quoted context omitted.

I'm an industrial electrician. A lot of poor electrical work is visible only to a fellow electrician, and sometimes only another industrial electrician. Bad technical work requires technical inspectors to criticize. Sometimes highly skilled ones.

I’ve reviewed a lot of papers, I don’t consider it the reviewers responsibility to manually verify all citations are real. If there was an unusual citation that was relied on heavily for the basis of the work, one would expect it to be checked. Things like broad prior work, you’d just assume it’s part of background. The reviewer is not a proofreader, they are checking the rigour and relevance of the work, which does…

As a reviewer I at least skimmed the papers for every reference in every paper that I review. If it isn't useful to furthering the point of the paper then my feedback is to remove the reference. Adding a bunch of junk because it is broadly related in a giant background section is a waste of everyone's time and should be removed. Most of the time you are mostly aware of the papers being cited anyway because that is the whole point of reviewing in your area of expertise.

Re: Over fifty new hallucinations in ICLR 2026 submissions

#357

Post by currymj »

I recommend actually clicking through and reading some of these papers.

Most of those I spot checked do not give an impression of high quality. Not just AI writing assistance but many seem to have AI-generated "ideas", often plausible nonsense. the reviewers often catch the errors and sometimes even the fake citations.

can I prove malfeasance beyond a reasonable doubt? no. but I personally feel quite confident many of the papers I checked are primarily AI-generated.

I feel really bad for any authors who submitted legitimate work but made an innocent mistake in their .bib and ended up on the same list as the rest of this stuff.

Re: Over fifty new hallucinations in ICLR 2026 submissions

#358

Post by alexcdot »

thfuran wrote: post #319
tpoacher wrote: post #271

Earlier quoted context omitted.

not OP, but that wouldn't really be necessary. One could submit their bibtex files and expect bibtex citations to be verifiable using a low level checker. Worst case scenario if your bibtex citation was a variant of one in the checker database you'd be asked to correct it to match the canonical version. However, as others here have stated, hallucinated "citations" are actually the lesser problem. Citing irrelevant pa…

Yes, I think verifying mere existence of the cited paper barely moves the needle. I mean, I guess automated verification of that is a cheap rejection criterion, but I don’t think it’s overall very useful.

really good point. one of the cofounders of gptzero here!

the tool gptzero used in the article also detects if the citation supports the claim too, if you scroll to "cited information accuracy" here: https://app.gptzero.me/documents/1641652a-c598-453f-9c94-e0b...

this is still in beta because its a much harder problem for sure, since its hard to determine if a 40 page paper supports a claims (if the paper claims X is computationally intractable, does that mean algorithms to compute approximate X are slow?)

Re: Over fifty new hallucinations in ICLR 2026 submissions

#359

Post by alexcdot »

bjourne wrote: post #182
grayhatter wrote: post #71

Earlier quoted context omitted.

> The reviewer is not a proofreader, they are checking the rigour and relevance of the work, which does not rest heavily on all of the references in a document. I've always assumed peer review is similar to diff review. Where I'm willing to sign my name onto the work of others. If I approve a diff/pr and it takes down prod. It's just as much my fault, no? > They are also assuming good faith. I can only relate this to…

For ICLR reviewers were asked to review 5 papers in two weeks. Unpaid voluntary work in addition to their normal teaching, supervision, meetings, and other research duties. It's just not possible to understand and thoroughly review each paper even for topic experts. If you want to compare peer review to coding, it's more like "no syntax errors, code still compiles" rather than pr review.

I really like what IJCAI is doing to pay reviewers to do this work, with the $100 fee from authors

Yeah its insane the workload reviewers are faced with + being an author who gets a review from a novice

Re: Over fifty new hallucinations in ICLR 2026 submissions

#360

Post by jqpabc123 »

ls612 wrote: post #150
jqpabc123 wrote: post #2

The legal system has a word to describe AI "slop" --- it is called "negligence". And as the remedy starts being applied (aka "liability"), the enthusiasm for AI will start to wane. I wouldn't be surprised if some businesses ban the use of AI --- starting with law firms.

The legal system has a word to describe software bugs --- it is called "negligence". And as the remedy starts being applied (aka "liability"), the enthusiasm for software will start to wane. What if anything do you think is wrong with my analogy? I doubt most people here support strict liability for bugs in code.

What if anything do you think is wrong with my analogy?

I think what is clearly wrong with your analogy is assuming that AI applies mostly to software and code production. This is actually a minor use-case for AI.

Government and businesses of all types ---doctors, lawyers, airlines, delivery companies, etc. are attempting to apply AI to uses and situations that can't be tested in advance the same way "vibe" code can. And some of the adverse results have already been ruled on in court.

https://www.evidentlyai.com/blog/ai-failures-examples

Post reply on HN