Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

gptzero.me

Post reply on HN
Print view View original

271–280 of 528 posts

Re: GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

#271

Post by gcr »

dataflow wrote: post #199
gcr wrote: post #150

Earlier quoted context omitted.

I'd personally like to see top conferences grow a "reproducibility" track. Each submission would be a short tech report that chooses some other paper to re-implement. Cap 'em at three pages, have a lightweight review process. Maybe there could be artifacts (git repositories, etc) that accompany each submission. This would especially help newer grad students learn how to begin to do this sort of research. Maybe doing…

Is it time for some sort of alternate degree to a PhD beyond a Master's? Showing, essentially, "this person can learn, implement, validate, and analyze the state of the art in this field"?

That sounds precisely like the function of a Ph.D. to me.

Re: GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

#272

Post by worik »

m-schuetz wrote: post #259
j2kun wrote: post #238

I spot-checked one of the flagged papers (from Google, co-authored by a colleague of mine) The paper was https://openreview.net/forum?id=0ZnXGzLcOg and the problem flagged was "Two authors are omitted and one (Kyle Richardson) is added. This paper was published at ICLR 2024." I.e., for one cited paper, the author list was off and the venue was wrong. And this citation was mentioned in the background section of the pa…

Bibtex are often also incorrectly generated. E.g., google scholar sometimes puts the names of the editors instead of the authors into the bibtex entry.

> Bibtex are often also incorrectly generated

...and including the erroneous entry is squarely the author's fault.

Papers should be carefully crafted, not churned out.

I guess that makes me sweetly naive

Re: GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

#273

Post by worik »

ls612 wrote: post #258
davidguetta wrote: post #254

Earlier quoted context omitted.

Yeah even the entire "Jane Doe / Jame Smith" my first thought is that it could have been a latex default value There was dumb stuff like this before the GPT era, it's far from convincing

There are people who just want to punish academics for the sake of punishing academics. Look at all the people downthread salivating over blacklisting or even criminally charging people who make errors like this with felony fraud. Its the perfect brew of anti AI and anti academia sentiment. Also, in my field (economics), by far the biggest source of finding old papers invalid (or less valid, most papers state multipl…

> I'd like to see the software engineers on this site say with a straight face that writing bugs should lead to jail time.

My hand is up.

I do not believe in gaol, but I do agree with the sentiment.

Re: GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

#274

Post by worik »

soiltype wrote: post #252
goalieca wrote: post #202

Earlier quoted context omitted.

Grad students don’t get to publish a thesis on reproduction. Everyone from the undergraduate research assistant to the tenured professor with research chairs are hyper focused on “publishing” as much “positive result” on “novel” work as possible

But that seems almost trivially solved. In software it's common to value independent verification - e.g. code review. Someone who is only focused on writing new code instead of careful testing, refactoring, or peer review is widely viewed as a shitty developer by their peers. Of course there's management to consider and that's where incentives are skewed, but we're talking about a different structure. Why wouldn't th…

> In software it's common to value independent verification - e.g. code review. Someone who is only focused on writing new code instead of careful testing, refactoring, or peer review is widely viewed as a shitty developer by their peers.

That is good practice

It is rare, not common. Managers and funders pay for features

Unreliable insecure software sells very well, so making reliable secure software is a "waste of money", generally

Re: GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

#275

Post by currymj »

rfrey wrote: post #220

There's a lot of good arguments in this thread about incentives: extremely convincing about why current incentives lead to exactly this behaviour, and also why creating better incentives is a very hard problem. If we grant that good carrots are hard to grow, what's the argument against leaning into the stick? Change university policies and processes so that getting caught fabricating data or submitting a paper with L…

the harsher the punishment, the more due process required.

i don't think there are any AI detection tools that are sufficiently reliable that I would feel comfortable expelling a student or ending someone's career based on their output.

for example, we can all see what's going on with these papers (and it appears to be even worse among ICLR submissions). but it is possible to make an honest mistake with your BibTeX. Or to use AI for grammar editing, which is widely accepted, and have it accidentally modify a data point or citation. There are many innocent mistakes which also count as plausible excuses.

in some cases further investigation maybe can reveal a smoking gun like fabricated data, which is academic misconduct whether done by hand or because an AI generated the LaTeX tables. punishments should be harsher for this than they are.

Re: GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

#276

Post by nearbuy »

j2kun wrote: post #238

I spot-checked one of the flagged papers (from Google, co-authored by a colleague of mine) The paper was https://openreview.net/forum?id=0ZnXGzLcOg and the problem flagged was "Two authors are omitted and one (Kyle Richardson) is added. This paper was published at ICLR 2024." I.e., for one cited paper, the author list was off and the venue was wrong. And this citation was mentioned in the background section of the pa…

The rate here (about 1% of papers) just doesn't seem that bad, especially if many of the errors are minor and don't affect the validity of the results. In other fields, over half of high-impact studies don't replicate.

Re: GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

#278

Post by currymj »

j2kun wrote: post #238

I spot-checked one of the flagged papers (from Google, co-authored by a colleague of mine) The paper was https://openreview.net/forum?id=0ZnXGzLcOg and the problem flagged was "Two authors are omitted and one (Kyle Richardson) is added. This paper was published at ICLR 2024." I.e., for one cited paper, the author list was off and the venue was wrong. And this citation was mentioned in the background section of the pa…

the earlier list of ICLR papers had way more egregious examples. Those were taken from the list of submissions not accepted papers however.

Re: GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

#279

Post by ls612 »

worik wrote: post #273
ls612 wrote: post #258

Earlier quoted context omitted.

There are people who just want to punish academics for the sake of punishing academics. Look at all the people downthread salivating over blacklisting or even criminally charging people who make errors like this with felony fraud. Its the perfect brew of anti AI and anti academia sentiment. Also, in my field (economics), by far the biggest source of finding old papers invalid (or less valid, most papers state multipl…

> I'd like to see the software engineers on this site say with a straight face that writing bugs should lead to jail time. My hand is up. I do not believe in gaol, but I do agree with the sentiment.

Let he who is without sin cast the first stone…

Re: GPTZero finds 100 new hallucinations in NeurIPS 2025 accepted papers

#280

Post by ls612 »

i_am_proteus wrote: post #268
j2kun wrote: post #238

I spot-checked one of the flagged papers (from Google, co-authored by a colleague of mine) The paper was https://openreview.net/forum?id=0ZnXGzLcOg and the problem flagged was "Two authors are omitted and one (Kyle Richardson) is added. This paper was published at ICLR 2024." I.e., for one cited paper, the author list was off and the venue was wrong. And this citation was mentioned in the background section of the pa…

>this error does make me pause to wonder how much of the rest of the paper used AI assistance And this is what's operative here. The error spotted, the entire class of error spotted, is easily checked/verified by a non-domain expert. These are the errors we can confirm readily, with obvious and unmistakable signature of hallucination. If these are the only errors, we are not troubled. However: we do not know if these…

Google scholar and the vagaries of copy/paste errors has mangled bibitex ever since it became a thing, a single citation with these sorts of errors may not even be AI, just “normal” mistakes.
Post reply on HN