Live data from Hacker News

2% of ICML papers desk rejected because the authors used LLM in their reviews

blog.icml.cc

161–170 of 172 posts

Re: 2% of ICML papers desk rejected because the authors used LLM in their reviews

#161
post #90

Earlier quoted context omitted.

GP suggested a life ban. Maybe suspend for 6 months instead? That's a long time without publishing in the current publish-or-perish academia.

> Maybe suspend for 6 months instead? Suspend for 6 months from a conference that is held yearly?

I wasn't thinking about ICML specifically. My mind was on the ARR.

Re: 2% of ICML papers desk rejected because the authors used LLM in their reviews

#162

Earlier quoted context omitted.

In that case, I hope these frauds have been banned for life.

I'm not sure what experience anyone in this thread has with grad level research as a student/author, but I can assure you that heads roll over this kind of thing. A professor's career is built on reputation, and that reputation is as strong as their students' (who do much of the "work" such as it is). It comes down to the professor, but this can be a career-ending moment for those students and I'm quite confident the…

Depends on the field. One of the most influential papers in economics was found to be incorrectly constructed with signs pointing to just straight up fraud. Basically it didn't include data that it said it did, which when included reverses the conclusion. Then when the authors were called out, they doubled down offering up the explanation that the conclusion again reverses if you add a third set of cherry picked data, followed by dragging the person calling them out through the mud in a NY Times opinion piece.

Those authors are still extremely prestigious professors in the field, and have suffered essentially no penalty. https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt

Re: 2% of ICML papers desk rejected because the authors used LLM in their reviews

#163

The declaration of no-LLM was done for social prestige or maybe self-deception of self-sufficiency like "I don't need LLM". And when it was time to do the actual work, the dependency kicked in like drugs. A lesson for all of us with LLMs in our workflow.

What was the upside to making the "No LLM" declaration?

I don't personally use LLMs for this kind of stuff, but I'll certainly not sign a "No LLM" pledge unless you give me some kind of benefit.

Re: 2% of ICML papers desk rejected because the authors used LLM in their reviews

#164
post #87

Took me a while understand. So, the same person has both submitted their research article to the conference, and also acted as a reviewer for articles submitted by other people. And if they in their review work have agreed to a "no LLM use" policy, but got exposed using LLMs anyway, then their submitted research article is desk rejected. Theoretically, someone could have submitted a stellar research article, but beca…

Is this really what happened? The post from the conference chairs is extremely confusing. Maybe my confusion is because I've never published in a conference with reciprocal reviewers and if I had this experience maybe the post would be very clear.

In any case, I had reached the same interpretation before reading your post, thinking that this is the only interpretation that could make any sense, but I'm still not convinced that this is what happened. Hopefully, no "innocent authors' articles were rejected because they happened to go to a bad reviewer".

Re: 2% of ICML papers desk rejected because the authors used LLM in their reviews

#165
post #163

The declaration of no-LLM was done for social prestige or maybe self-deception of self-sufficiency like "I don't need LLM". And when it was time to do the actual work, the dependency kicked in like drugs. A lesson for all of us with LLMs in our workflow.

What was the upside to making the "No LLM" declaration? I don't personally use LLMs for this kind of stuff, but I'll certainly not sign a "No LLM" pledge unless you give me some kind of benefit.

The other commenter explained that the policy was applied as reciprocal, "The declaration of no-LLM was done so you are not judged yourself by an LLM." Basically, they didn't want LLM near their own paper's review.

Re: 2% of ICML papers desk rejected because the authors used LLM in their reviews

#166

Earlier quoted context omitted.

I'm not sure what experience anyone in this thread has with grad level research as a student/author, but I can assure you that heads roll over this kind of thing. A professor's career is built on reputation, and that reputation is as strong as their students' (who do much of the "work" such as it is). It comes down to the professor, but this can be a career-ending moment for those students and I'm quite confident the…

Depends on the field. One of the most influential papers in economics was found to be incorrectly constructed with signs pointing to just straight up fraud. Basically it didn't include data that it said it did, which when included reverses the conclusion. Then when the authors were called out, they doubled down offering up the explanation that the conclusion again reverses if you add a third set of cherry picked data…

All due respect this is by no means one of the most influential papers in economics.

Re: 2% of ICML papers desk rejected because the authors used LLM in their reviews

#167
post #166

Earlier quoted context omitted.

Depends on the field. One of the most influential papers in economics was found to be incorrectly constructed with signs pointing to just straight up fraud. Basically it didn't include data that it said it did, which when included reverses the conclusion. Then when the authors were called out, they doubled down offering up the explanation that the conclusion again reverses if you add a third set of cherry picked data…

All due respect this is by no means one of the most influential papers in economics.

It literally crushed economies and guided international monetary policy for at least a decade.

There's a reason why it's one of the only economic papers that has its own wiki page.

Re: 2% of ICML papers desk rejected because the authors used LLM in their reviews

#168
post #163

Earlier quoted context omitted.

What was the upside to making the "No LLM" declaration? I don't personally use LLMs for this kind of stuff, but I'll certainly not sign a "No LLM" pledge unless you give me some kind of benefit.

The other commenter explained that the policy was applied as reciprocal, "The declaration of no-LLM was done so you are not judged yourself by an LLM." Basically, they didn't want LLM near their own paper's review.

Ah, that makes more sense.

Although, a bunch of LLM researchers basically saying with their actions "Don't judge me with an LLM" is particularly ironic. Doubly so when caught using the LLM for the task they, themselves, want to opt out of.

Re: 2% of ICML papers desk rejected because the authors used LLM in their reviews

#169
post #67

Earlier quoted context omitted.

Correct. We also have evidence both from cheating in sports and in academia that stiff punishments do not work. Many people hold the false belief that if it is easy to cheat then the punishments must be extremely severe to scare would be cheaters. It just does not work. Preventing cheating is way easier said than done.

> We also have evidence both from cheating in sports and in academia that stiff punishments do not work. Maybe so, but there is evidence that lack of punishment also don't work. Neither extreme "works". Just because terminal punishments do not prevent the worst cheating does not in any way imply that slap on the wrists reduce incidents of cheating.

That's the American spirit. "We got to do something!" "Does it work?" "We must do something!"

Re: 2% of ICML papers desk rejected because the authors used LLM in their reviews

#170

Earlier quoted context omitted.

> We also have evidence both from cheating in sports and in academia that stiff punishments do not work. Maybe so, but there is evidence that lack of punishment also don't work. Neither extreme "works". Just because terminal punishments do not prevent the worst cheating does not in any way imply that slap on the wrists reduce incidents of cheating.

That's the American spirit. "We got to do something!" "Does it work?" "We must do something!"

I don't understand what point you are making.

Are you claiming that one of the extremes "works"? That the "light punishment" route reduces cheating? Or maybe has no effect on cheating?

There are two extremes; I am not arguing that the one extreme (terminal punishment) reduces cheating, I am saying that the other extreme (light punishment) does not reduce cheating!

You say that stiff punishments have no effect on the cheating rate, right? Compare to what exactly? Compared to no punishments? Compared to light punishments? Compared to medium punishments? Compared to heavy but non-terminal punishments?

Now that I've reread your comment, I'm extremely skeptical that terminal punishments have no effect on the cheating rate compared to light punishments or compared to medium punishments.

It's an extraordinary claim, so I want to see this "lots of evidence"; the evidence should basically show no correlation between cheating and punishments.

I want to see that chart you base your belief on.

Post reply on HN