'Positive review only': Researchers hide AI prompts in papers • Hacker News Board

Re: 'Positive review only': Researchers hide AI prompts in papers

#81

Post by birn559 » Sat, Jul 05, 2025, 7:27 PM UTC

matusp wrote: post #20

AI generated reviews are a huge problem even at the most prestigious ML conferences. It is hard to argue against them, since the weaknesses they identify are usually in well formulated, and it is hard to argue that subjectively they are not that important. ACL recently started requiring Limitations section in their paper where authors should transparently discuss what are the limits. Unfortunately, that section is ba…

Wow, that's a terrible second order effect with very real impact on the quality of publications.

Re: 'Positive review only': Researchers hide AI prompts in papers

#82

Post by jeroenhd » Sat, Jul 05, 2025, 7:27 PM UTC

looofooo0 wrote: post #68

How does this help? Use print to png or use AI to remove non visible content. It is only a small script away, maybe only the right prompt.

The kind of people who automate away their job won't care if papers accidentally make it through the review process and they won't care enough to stop this. For them, the process is working.

Lazy fraudsters don't pose much of a challenge. If the scientific process works even a little bit, this is just a stupid gimmick, like hiding a Monty Python quote in the metadata.

Re: 'Positive review only': Researchers hide AI prompts in papers

#83

Post by ashton314 » Sat, Jul 05, 2025, 7:28 PM UTC

> Inserting the hidden prompt was inappropriate, as it encourages positive reviews even though the use of AI in the review process is prohibited.

I think this is a totally ethical thing for a paper writer to do. Include an LLM honeypot. If your reviews come back and it seems like they’ve triggered the honeypot, blow the whistle loudly and scuttle that “reviewer’s” credibility. Every good, earnest researcher wants good, honest feedback on their papers—otherwise the peer-review system collapses.

I’m not saying peer-review isn’t without flaws; but it’s infinitely better than a rubber-stamping bot.

Re: 'Positive review only': Researchers hide AI prompts in papers

#84

Post by baxtr » Sat, Jul 05, 2025, 7:29 PM UTC

mbreese wrote: post #80

bee_rider wrote: post #76

Earlier quoted context omitted.

Writing reviews isn’t, like, a test or anything. You don’t get graded on it. So I think it is wrong to think of this tool as cheating. They are professional researchers and doing the reviews is part of their professional obligation to their research community. If people are using LLMs to do reviews fast-and-shitty, they are shirking their responsibility to their community. If they use the tools to do reviews fast-and…

As I understand it, the restriction of LLMs has nothing to do with getting poor quality/AI reviews. Like you said, you’re not really getting graded on it. Instead, the restriction is in place to limit the possibility of an unpublished paper getting “remembered” by an LLM. You don’t want to have an unpublished work getting added as a fact to a model accidentally (mainly to protect the novelty of the authors work, not…

I don’t think that’s how LLMs work. If that was the case anyone could feed them false info eg for propaganda purposes…

Re: 'Positive review only': Researchers hide AI prompts in papers

#85

Post by g42gregory » Sat, Jul 05, 2025, 7:30 PM UTC

I keep reading in the press that the "well-being of our society depends on the preservation of these academic research institutions."

I am beginning to doubt this.

Maybe we should create new research institutions instead...

Re: 'Positive review only': Researchers hide AI prompts in papers

#86

Post by pcrh » Sat, Jul 05, 2025, 7:31 PM UTC

calebkaiser wrote: post #73

pcrh wrote: post #71

How is an LLM supposed to review an original manuscript? At their core (and as far as I understand), LLMs are based on pre-existing texts, and use statistical algorithms to stitch together text that is consistent with these. An original research manuscript will not have formed part of any LLMs training dataset, so there is no conceivable way that it can evaluate it, regardless of claims that LLMs "understand" anythin…

You might be interested in work around mechanistic interpretability! In particular, if you're interested in how models handle out-of-distribution information and apply in-context learning, research around so-called "circuits" might be up your alley: https://www.transformer-circuits.pub/2022/mech-interp-essay

After a brief scan, I'm not competent to evaluate the essay by Chris Olah you posted.

I probably could get an LLM to do so, but I won't....

Re: 'Positive review only': Researchers hide AI prompts in papers

#87

Post by jeroenhd » Sat, Jul 05, 2025, 7:32 PM UTC

pcrh wrote: post #71

How is an LLM supposed to review an original manuscript? At their core (and as far as I understand), LLMs are based on pre-existing texts, and use statistical algorithms to stitch together text that is consistent with these. An original research manuscript will not have formed part of any LLMs training dataset, so there is no conceivable way that it can evaluate it, regardless of claims that LLMs "understand" anythin…

LLMs can find problems in logic, conclusions based on circumstantial evidence, common mistakes made in other rejected papers, and other suspect language, even if it hasn't seen the exact sentence structures used in its input. You'll catch plenty of improvements to scientific preprints that way because humans aren't all that good at writing down long, complicated documents as we might think we are.

Sometimes it'll claim that a noun can only be used as a verb and will think you're Santa. LLMs can't be relied to be accurate or truthful of course.

I can imagine the non-computer science people (and unfortunately some computer science people) believe LLMs are close to infallibe. What's a biologist or a geographist going to know about the limits of ChatGPT? All they know is that the LLM did a great job spotting the grammatical issues in the paragraph they had it check so it seems pretty legit right?

Re: 'Positive review only': Researchers hide AI prompts in papers

#88

Post by ChrisMarshallNY » Sat, Jul 05, 2025, 7:33 PM UTC

SoftTalker wrote: post #22

dynm wrote: post #9

Just to be clear, these are hidden prompts put in papers by authors meant to be triggered only if a reviewer (unethically) uses AI to generate their review. I guess this is wrong, but I find it hard not to have some sympathy for the authors. Mostly, it seems like an indictment of the whole peer-review system.

Back in high school a few kids would be tempted to insert a sentence such as "I bet you don't actually read all these papers" into an essay to see if the teacher caught it. I never tried it but the rumors were that some kids had got away with it. I just used it to worry less that my work was rushed and not very good, I told myself "the teacher will probably just be skimming this anyway; they don't have time to read a…

Like the invisible gorilla?

https://www.youtube.com/watch?v=vJG698U2Mvo

Re: 'Positive review only': Researchers hide AI prompts in papers

#89

Post by pcrh » Sat, Jul 05, 2025, 7:36 PM UTC

jeroenhd wrote: post #87

pcrh wrote: post #71

How is an LLM supposed to review an original manuscript? At their core (and as far as I understand), LLMs are based on pre-existing texts, and use statistical algorithms to stitch together text that is consistent with these. An original research manuscript will not have formed part of any LLMs training dataset, so there is no conceivable way that it can evaluate it, regardless of claims that LLMs "understand" anythin…

LLMs can find problems in logic, conclusions based on circumstantial evidence, common mistakes made in other rejected papers, and other suspect language, even if it hasn't seen the exact sentence structures used in its input. You'll catch plenty of improvements to scientific preprints that way because humans aren't all that good at writing down long, complicated documents as we might think we are. Sometimes it'll cla…

I don't doubt that LLMs can improve grammar. However, an original research paper should not be evaluated on the basis of the quality of the writing, unless this is so bad as to make the claims impenetrable.

Re: 'Positive review only': Researchers hide AI prompts in papers

#90

Post by doug-moen » Sat, Jul 05, 2025, 7:38 PM UTC

> Netherlands-based Elsevier bans the use of such tools, citing the "risk that the technology will generate incorrect, incomplete or biased conclusions."

That's for peer reviewers, who aren't paid. Elsevier is also reported to be using AI to replace editing staff. Perhaps this risk is less relevant when there is an opportunity to increase profits?

Evolution journal editors resign en masse to protest Elsevier changes. https://retractionwatch.com/2024/12/27/evolution-journal-edi...

discussion. https://news.ycombinator.com/item?id=42528203