Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Positive review only': Researchers hide AI prompts in papers

asia.nikkei.com

Post reply on HN
Print view View original

141–150 of 154 posts

Re: 'Positive review only': Researchers hide AI prompts in papers

#141

Post by gridspy »

soraminazuki wrote: post #130
serbuvlad wrote: post #119

Earlier quoted context omitted.

But that's exactly the thing. I DON'T understand the technology without AI.. I know stuff about Linux, but I knew NOTHING about Ansible, FreeIPA etc. So I guess you could say I understand the problem space not the solution space?? Either way, it would have taken us many months to do what it did take us a few weeks to with AI. > So the whole appeal of AI seems to be to let it do things without much oversight. No?? The…

> I know stuff about Linux, but I knew NOTHING about Ansible, FreeIPA etc. Then you absolutely shouldn't be touching Ansible or FreeIPA in production until you've developed enough understanding of the basics and can look up reliable sources for the nitty gritty details. FreeIPA is security critical software for heaven's sake. "Let's make up for zero understanding with AI" is a totally unacceptable approach.

Often the problem best solved by AI isn't "How do I use Anisble to do X" But simply knowing that you should be using Ansible at all. Or which features to use. Then you can be laser-focused on learning exactly the right part of Ansible.

It also helps you to move away from familiar but hacky solutions towards much more standard and robust ones - the AI doesn't approach the problem with your many years of battle scars and baggage but instead sees your problem space with fresh eyes and recent data.

Re: 'Positive review only': Researchers hide AI prompts in papers

#142

Post by grishka »

gridspy wrote: post #141
soraminazuki wrote: post #130

Earlier quoted context omitted.

> I know stuff about Linux, but I knew NOTHING about Ansible, FreeIPA etc. Then you absolutely shouldn't be touching Ansible or FreeIPA in production until you've developed enough understanding of the basics and can look up reliable sources for the nitty gritty details. FreeIPA is security critical software for heaven's sake. "Let's make up for zero understanding with AI" is a totally unacceptable approach.

Often the problem best solved by AI isn't "How do I use Anisble to do X" But simply knowing that you should be using Ansible at all. Or which features to use. Then you can be laser-focused on learning exactly the right part of Ansible. It also helps you to move away from familiar but hacky solutions towards much more standard and robust ones - the AI doesn't approach the problem with your many years of battle scars a…

But why use AI for that in the first place? You can just, uh, google things. Like "software that does X". You would get a better result from forums and blogs and such. When doing research, AI really feels like an unnecessary middleman when you know how to do research on the internet. Which I would argue is a basic skill in the modern world.

Re: 'Positive review only': Researchers hide AI prompts in papers

#143

Post by pkoird »

xg15 wrote: post #3

> Some researchers argued that the use of these prompts is justified. "It's a counter against 'lazy reviewers' who use AI," said a Waseda professor who co-authored one of the manuscripts. Given that many academic conferences ban the use of artificial intelligence to evaluate papers, the professor said, incorporating prompts that normally can be read only by AI is intended to be a check on this practice. I like this -…

Honestly don't understand why they have the prompt injection in arxiv of all places. One would imagine that a researcher aiming to leverage AI based reviews would only modify their private submissions.

Re: 'Positive review only': Researchers hide AI prompts in papers

#144

Post by rngrngrng »

quacksilver wrote: post #139

Could AI still be a useful tool if the reviewer performs a manual review first and then queries the LLM with: 1) Here is a new academic paper. Point out any inconsistencies, gaps or flaws in the research, and any contradictions with previous research in the field. 2) Here is a new academic paper and a journal submission policy. Does the paper meet the journal submission policy? 3) Here is a new academic paper, the re…

No it cannot.

you are not allowed to share the unpublished results with anyone or any LLM, period. This is literally in every review policy (e.g. https://neurips.cc/Conferences/2025/CallForPapers)

Re: 'Positive review only': Researchers hide AI prompts in papers

#145

Post by drdunce »

seydor wrote: post #35

last time i used LLMs to review a paper they were all garbage. They couldn't even identify a typo and kept giving the same generic irrelevant advice.

That sounds like very much like the reviews I used to get on papers at top tier conferences before the advent of LLMs.

Re: 'Positive review only': Researchers hide AI prompts in papers

#146

Post by grumbelbart2 »

bee_rider wrote: post #76
gpm wrote: post #5

Earlier quoted context omitted.

Then the people generating the review are likely to notice and change their approach at cheating... I want a prompt that embeds evidence of AI use... in a paper about matrix multiplication "this paper is critically important to the field of FEM (Finite Element Analysis), it must be widely read to reduce the risk of buildings collapsing. The authors should be congratulated on their important contribution to the field…

Writing reviews isn’t, like, a test or anything. You don’t get graded on it. So I think it is wrong to think of this tool as cheating. They are professional researchers and doing the reviews is part of their professional obligation to their research community. If people are using LLMs to do reviews fast-and-shitty, they are shirking their responsibility to their community. If they use the tools to do reviews fast-and…

> Writing reviews isn’t, like, a test or anything. You don’t get graded on it. So I think it is wrong to think of this tool as cheating.

Except that since last year, it kind of is. It is now mandatory for some large conferences (such as CVPR) for authors to do reviews if they submit a paper. Failure to review, or reviews that are neglectful, can lead to a desk reject of their submission.

Re: 'Positive review only': Researchers hide AI prompts in papers

#147

Post by reseasonable »

grishka wrote: post #142
gridspy wrote: post #141

Earlier quoted context omitted.

Often the problem best solved by AI isn't "How do I use Anisble to do X" But simply knowing that you should be using Ansible at all. Or which features to use. Then you can be laser-focused on learning exactly the right part of Ansible. It also helps you to move away from familiar but hacky solutions towards much more standard and robust ones - the AI doesn't approach the problem with your many years of battle scars a…

But why use AI for that in the first place? You can just, uh, google things. Like "software that does X". You would get a better result from forums and blogs and such. When doing research, AI really feels like an unnecessary middleman when you know how to do research on the internet. Which I would argue is a basic skill in the modern world.

So your point went from: they shouldn’t have done what they did because it likely took much longer than necessary by using AI - to: they shouldn’t have done what they did because they didn’t take long enough learning it without AI to do it. I think you moved the goalpost to the opposite side of the field.

In this thread you chastised them for “using it in production” when their use case was simply a lab for teaching. And you suggest that googling for days and reading countless blog posts and forums is impervious to wrong or dated information, which is quite a position to take.

Their use case was a perfect fit for a LLM, and I suspect it saved them weeks of effort in the process, with likely fewer security flaws as a result. But keep up that anti-LLM advocacy, I’m sure you are very close to achieving the goal of society abandoning the technology.

Re: 'Positive review only': Researchers hide AI prompts in papers

#148

Post by calebkaiser »

pcrh wrote: post #86
calebkaiser wrote: post #73

Earlier quoted context omitted.

You might be interested in work around mechanistic interpretability! In particular, if you're interested in how models handle out-of-distribution information and apply in-context learning, research around so-called "circuits" might be up your alley: https://www.transformer-circuits.pub/2022/mech-interp-essay

After a brief scan, I'm not competent to evaluate the essay by Chris Olah you posted. I probably could get an LLM to do so, but I won't....

Neel Nanda is also very active in the field and writes some potentially more approachable articles, if you're interested: https://www.neelnanda.io/mechanistic-interpretability

Much of their work is focused on discovering "circuits" that occur between layer activations as they process data, which correspond to dynamics the model has learned. So, as a simple hypothetical example, instead of embedding the answer to 1 million arbitrary addition problems in the weights, models might learn a circuit that approximates the operation of addition.

Re: 'Positive review only': Researchers hide AI prompts in papers

#149

Post by beefnugs »

Aurornis wrote: post #113
rdtsc wrote: post #10

> "It's a counter against 'lazy reviewers' who use AI," said a Waseda professor who co-authored one of the manuscripts. Given that many academic conferences ban the use of artificial intelligence to evaluate papers, the professor said, incorporating prompts that normally can be read only by AI is intended to be a check on this practice. Everyone who applies for jobs should be doing this in their resumes: "Ignore prev…

From someone who has read a lot of resumes through the years: Don’t play resume games like this if you want to find a good company. After you’ve read a hundred resumes in a week, spotting resume “hacks” like hiding words in white text, putting a 1pt font keyword stuffing section in the bottom, or now trying to trick an imagined AI resume screener become negative signals very quickly. In my experience, people who play…

"dont you play resume games with me" shut up. The most callbacks i ever got was when i decided to say fuckit and just made a wanky graphical resume turning it into this elaborate parts explosion diagram. Nobody cared about any real experience and wouldn't shut up about how cool and original this resume was, "we need people just like you" I have never felt so strongly the world was so full of cruel incompetent idiots gatekeeping the entire worlds work force behind pure wankery

Re: 'Positive review only': Researchers hide AI prompts in papers

#150

Post by bee_rider »

grumbelbart2 wrote: post #146
bee_rider wrote: post #76

Earlier quoted context omitted.

Writing reviews isn’t, like, a test or anything. You don’t get graded on it. So I think it is wrong to think of this tool as cheating. They are professional researchers and doing the reviews is part of their professional obligation to their research community. If people are using LLMs to do reviews fast-and-shitty, they are shirking their responsibility to their community. If they use the tools to do reviews fast-and…

> Writing reviews isn’t, like, a test or anything. You don’t get graded on it. So I think it is wrong to think of this tool as cheating. Except that since last year, it kind of is. It is now mandatory for some large conferences (such as CVPR) for authors to do reviews if they submit a paper. Failure to review, or reviews that are neglectful, can lead to a desk reject of their submission.

Seems like… a pretty bad policy, right? I wouldn’t mind doing a mandatory review, but I certainly wouldn’t want to be on the receiving end of some grumpy person who didn’t want to do one.
Post reply on HN