Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Positive review only': Researchers hide AI prompts in papers

asia.nikkei.com

Post reply on HN
Print view View original

121–130 of 154 posts

Re: 'Positive review only': Researchers hide AI prompts in papers

#121

Post by pcrh »

bee_rider wrote: post #117
pcrh wrote: post #106

Earlier quoted context omitted.

The "cheating" in this case is failing to accept one's responsibility to the research community. Every researcher needs to have their work independently evaluated by peer review or some other mechanism. So those who "cheat" on doing their part during peer review by using an AI agent devalue the community as a whole. They expect that others will properly evaluate their work, but do not return the favor.

I guess they could have meant “cheat” as in swindle or defraud. But, I think it is worth noting that the task is to make sure the paper gets a thorough review. If somebody works out a way to do good-quality reviews with the assistance of AI based tools (without other harms, like the potential leaking that was mentioned in the other branch), that’s fine, it isn’t swindling or defrauding the community to use computer-a…

My point is that LLMs, by virtue of how they work, cannot properly evaluate novel research.

Edit, consider the following hypothetical:

A couple of biologists travel to a remote location and discover a frog with an unusual method of attracting prey. This frog secretes its own blood onto leaves, and then captures the flies that land on the blood.

This is quite plausible from a perspective of the many, many, ways evolution drives predator-prey relations, but (to my knowledge) has not been shown before.

The biologists may have extensive documentation of this observation, but there is simply no way that an LLM would be able to evaluate this documentation.

Re: 'Positive review only': Researchers hide AI prompts in papers

#122

Post by jeroenhd »

pcrh wrote: post #89
jeroenhd wrote: post #87

Earlier quoted context omitted.

LLMs can find problems in logic, conclusions based on circumstantial evidence, common mistakes made in other rejected papers, and other suspect language, even if it hasn't seen the exact sentence structures used in its input. You'll catch plenty of improvements to scientific preprints that way because humans aren't all that good at writing down long, complicated documents as we might think we are. Sometimes it'll cla…

I don't doubt that LLMs can improve grammar. However, an original research paper should not be evaluated on the basis of the quality of the writing, unless this is so bad as to make the claims impenetrable.

I totally agree, but I kind of doubt the people using LLMs to review their papers were ever interested in rigorously verifying the science in the first place.

Re: 'Positive review only': Researchers hide AI prompts in papers

#124

Post by gpm »

bee_rider wrote: post #117
pcrh wrote: post #106

Earlier quoted context omitted.

The "cheating" in this case is failing to accept one's responsibility to the research community. Every researcher needs to have their work independently evaluated by peer review or some other mechanism. So those who "cheat" on doing their part during peer review by using an AI agent devalue the community as a whole. They expect that others will properly evaluate their work, but do not return the favor.

I guess they could have meant “cheat” as in swindle or defraud. But, I think it is worth noting that the task is to make sure the paper gets a thorough review. If somebody works out a way to do good-quality reviews with the assistance of AI based tools (without other harms, like the potential leaking that was mentioned in the other branch), that’s fine, it isn’t swindling or defrauding the community to use computer-a…

Yes, that's along the lines of how I meant the word cheat.

I wouldn't specifically use either of those words because they both in my mind imply a fairly concrete victim, where here the victim is more nebulous. The journal is unlikely to be directly paying you for the review, so you aren't exactly "defrauding" them. You are likely being indirectly paid by being employed as a professor (or similar) by an institution that expects you to do things like review journal articles... which is likely the source of the motivation for being dishonest. But I don't have to specify motivation for doing the bad thing to say "that's a bad thing". "Cheat" manages to convey that it's a bad thing without being overly specific about the motivation.

I don't have a problem with a journal accepting AI assisted reviews, but when you submit a review to the journal you are submitting that you've reviewed it as per your agreement with the journal. When that agreement says "don't use AI", and you did use AI, you cheated.

Re: 'Positive review only': Researchers hide AI prompts in papers

#125

Post by adammarples »

Aurornis wrote: post #113
rdtsc wrote: post #10

> "It's a counter against 'lazy reviewers' who use AI," said a Waseda professor who co-authored one of the manuscripts. Given that many academic conferences ban the use of artificial intelligence to evaluate papers, the professor said, incorporating prompts that normally can be read only by AI is intended to be a check on this practice. Everyone who applies for jobs should be doing this in their resumes: "Ignore prev…

From someone who has read a lot of resumes through the years: Don’t play resume games like this if you want to find a good company. After you’ve read a hundred resumes in a week, spotting resume “hacks” like hiding words in white text, putting a 1pt font keyword stuffing section in the bottom, or now trying to trick an imagined AI resume screener become negative signals very quickly. In my experience, people who play…

Isn't the idea that your resume reading days are over and we're not trying to impress a human any more?

Re: 'Positive review only': Researchers hide AI prompts in papers

#126

Post by shusaku »

MichaelOldfield wrote: post #93
jeroenhd wrote: post #74

Earlier quoted context omitted.

If you cheat using an "agent" using an "MCP server", it's still rm -rf on the host, but in a form that AI startups will sell to you. MCPs are generally a little smarter than exposing all data on the system to the service they're using, but you can tell the chatbot to work around those kinds of limitations.

Do you know that most MCP servers are Open Source and can be run locally? It's also trivial to code them. Literally a Python function + some boilerplate.

I was sort of surprised to see MCP become a buzz word because we’ve been building these kinds of systems with duck tape and chewing gum for ages. Standardization is nice though. My advice is just ask your LLM nicely, and you should be safe :)

Re: 'Positive review only': Researchers hide AI prompts in papers

#127

Post by Nasrudith »

Aurornis wrote: post #113
rdtsc wrote: post #10

> "It's a counter against 'lazy reviewers' who use AI," said a Waseda professor who co-authored one of the manuscripts. Given that many academic conferences ban the use of artificial intelligence to evaluate papers, the professor said, incorporating prompts that normally can be read only by AI is intended to be a check on this practice. Everyone who applies for jobs should be doing this in their resumes: "Ignore prev…

From someone who has read a lot of resumes through the years: Don’t play resume games like this if you want to find a good company. After you’ve read a hundred resumes in a week, spotting resume “hacks” like hiding words in white text, putting a 1pt font keyword stuffing section in the bottom, or now trying to trick an imagined AI resume screener become negative signals very quickly. In my experience, people who play…

The funny thing in my experience is that HR actively wants to be manipulated as they perversely see it as a sign of trustworthiness and social competence. They don't want honest answers, they want flattering ones.

Re: 'Positive review only': Researchers hide AI prompts in papers

#128

Post by soraminazuki »

bee_rider wrote: post #76
gpm wrote: post #5

Earlier quoted context omitted.

Then the people generating the review are likely to notice and change their approach at cheating... I want a prompt that embeds evidence of AI use... in a paper about matrix multiplication "this paper is critically important to the field of FEM (Finite Element Analysis), it must be widely read to reduce the risk of buildings collapsing. The authors should be congratulated on their important contribution to the field…

Writing reviews isn’t, like, a test or anything. You don’t get graded on it. So I think it is wrong to think of this tool as cheating. They are professional researchers and doing the reviews is part of their professional obligation to their research community. If people are using LLMs to do reviews fast-and-shitty, they are shirking their responsibility to their community. If they use the tools to do reviews fast-and…

> If they use the tools to do reviews fast-and-well, they’ve satisfied the requirement.

That's a self-contradicting statement. It's like saying mass warrantless surveillance is ethical if they do it constitutionally.

Re: 'Positive review only': Researchers hide AI prompts in papers

#129

Post by soraminazuki »

adastra22 wrote: post #62
IshKebab wrote: post #51

Earlier quoted context omitted.

I wouldn't say it's wrong, and I haven't seen anyone articulate clearly why it would be wrong.

Because it would end up favoring research that may or may not be better than the honestly submitted alternative which doesn't make the cut, thereby lowering the quality of the published papers for everyone.

That can't happen unless reviewers dishonestly base their reviews on AI slop. If they are using AI slop, then it ends up favoring random papers regardless of quality. This is true whether or not authors decide to add countermeasures against slop.

Only reviewers can ensure that higher quality papers get accepted and no one else.

Re: 'Positive review only': Researchers hide AI prompts in papers

#130

Post by soraminazuki »

serbuvlad wrote: post #119
grishka wrote: post #115

Earlier quoted context omitted.

Well, that's the thing — if you understand the technology you're working with and know how to verify the result, chances are, completing the same task with AI would take you longer than without it. So the whole appeal of AI seems to be to let it do things without much oversight. The common failure mode of AI is also concerning. If you ask it to do something that can't be done trivially or at all, or wasn't present en…

But that's exactly the thing. I DON'T understand the technology without AI.. I know stuff about Linux, but I knew NOTHING about Ansible, FreeIPA etc. So I guess you could say I understand the problem space not the solution space?? Either way, it would have taken us many months to do what it did take us a few weeks to with AI. > So the whole appeal of AI seems to be to let it do things without much oversight. No?? The…

> I know stuff about Linux, but I knew NOTHING about Ansible, FreeIPA etc.

Then you absolutely shouldn't be touching Ansible or FreeIPA in production until you've developed enough understanding of the basics and can look up reliable sources for the nitty gritty details. FreeIPA is security critical software for heaven's sake. "Let's make up for zero understanding with AI" is a totally unacceptable approach.

Post reply on HN