Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-fabricated scientific papers on Google Scholar

misinforeview.hks.harvard.edu

Post reply on HN
Print view View original

71–80 of 107 posts

Re: GPT-fabricated scientific papers on Google Scholar

#71

Post by bravura »

Der_Einzige wrote: post #47
matusp wrote: post #30

Earlier quoted context omitted.

We already got an LLM generated meta review that was very clearly just summarization of reviews. There were some pretty egregious cases of borderline hallucinated remarks. This was ACL Rolling Review, so basically the most prestigious NLP venue and the editors told us to suck it up. Very disappointing and I genuinely worry about the state of science and how this will affect people who rely on scientometric criteria.

With NeurIPS 2024 reviews going on right now, I'm sure that a whole lot of these kind of reviews are being generated daily.

With ICLR paper deadline coming up, I guess it's worth wargaming how GPT4 would review my submission.

Re: GPT-fabricated scientific papers on Google Scholar

#72

Post by joshvm »

jll29 wrote: post #61
matusp wrote: post #30

Earlier quoted context omitted.

We already got an LLM generated meta review that was very clearly just summarization of reviews. There were some pretty egregious cases of borderline hallucinated remarks. This was ACL Rolling Review, so basically the most prestigious NLP venue and the editors told us to suck it up. Very disappointing and I genuinely worry about the state of science and how this will affect people who rely on scientometric criteria.

Most conferences have been flooded with submissions, and ACL is no exception. A consequence of that is that there are not sufficient numbers of reviewers available who are qualified to review these manuscripts. Conference organizers might be keen to accept many or most who offer to volunteer, but clearly there is now a large pool of people that have never done this before, and were never taught how to do this. Add so…

We had what we strongly suspect is an LLM-written review for NeurIPS. It was kind of subtle if you weren't looking carefully and I can see that an AC might miss it. The suggestions for improvement weren't _wrong_, but the GPT response picked up on some extremely specific things in the paper that were mostly irrelevant (other reviewers actually pointed out the odd typo and small corrections or improvemnts where we'd made statements).

Pretty hard to combat. We just rebutted as if it were a real review - maybe it was - and hope that the chairs see it. Speaking to other folks, opinions are split over whether this sort of review should be flagged. I know some people who tried to query a review and it didn't help.

There were other small cues - the English was perfect, while other reviewers made small slips indicative of non-native speakers. One was simply the discrepancy between the tone of the review (generally very positive) and the middle-of-the-road rating and confidence. The structure of the review was very "The authors do X, Y, Z. This is important because A, B, C." and the reviewer didn't bother to fill out any of the other review sections (they just wrote single-word answeres to all of them).

The kicker was actually putting our paper in to 4o and asking it to write a review and seeing the same keywords pop up.

Re: GPT-fabricated scientific papers on Google Scholar

#73

Post by croes »

nomilk wrote: post #5

GPT might make fabricating scientific papers easier, but let's not forget how many humans fabricated scientific research in recent years - they did a great job without AI! For any who haven't seen/heard, this makes for some entertaining and eye-opening viewing! https://www.youtube.com/results?search_query=academic+fraud

But AI is to papers what the assembly line was to cars.

Re: GPT-fabricated scientific papers on Google Scholar

#74

Post by ben_w »

Eisenstein wrote: post #66
basch wrote: post #56

Earlier quoted context omitted.

It might follow to say that current LLM;s arent trained to generate papers, BUT they also don't really need to reason. They just need to mimic the appearance of reason, follow the same pattern of progression. Ingesting enough of what amounts to executed templates will teach it to generate its own results as if output from the same template.

What is the difference between 'reasoning' and 'appearing to be reasoning' if the results are the same with the same input?

The outputs aren't really the same, they simply seem plausible at first glance.

For example, I recently experimented with using ChatGPT to translate a Wikipedia article, on the grounds that it mighy maintain all the formatting and that Transformer models are also used by Google Translate.

As it was an experiment, I did actually check the results before submitting the translated article.

First roughly 3/4 were fine. Final quarter was completely invented but plausible, including references.

LLMs are very useful tools, I'll gladly use them to help with various tasks and they can (with low reliability but it has happened) even manage a whole project, but right now they should treated with caution and not left unsupervised — Peter principle, being promoted beyond their competence, still applies even though they're not human employees.

Re: GPT-fabricated scientific papers on Google Scholar

#75

Post by joshvm »

bravura wrote: post #71
Der_Einzige wrote: post #47

Earlier quoted context omitted.

With NeurIPS 2024 reviews going on right now, I'm sure that a whole lot of these kind of reviews are being generated daily.

With ICLR paper deadline coming up, I guess it's worth wargaming how GPT4 would review my submission.

See my other post - we had exactly this for NeurIPS. It is definitely worth seeing what GPT says about your paper if only because it's a free review. The criticisms it gave us weren't wrong per se, they were just weakly backed up and it would still be up to a reviewer to judge how relevant they are or not. Every paper has downsides, but you need domain knowledge to judge if it's a small issue or a killer. Amusingly, our LLM-reviewer gave a much lower score than when we asked GPT to provide a rating (and also significantly lower than the other reviewers).

One example was that GPT took an explicit geographic location from a figure caption and used that as a reference point when suggesting improvements (along the lines of "location X is under-represented on this map") I assume because it places some high degree of relevance to figures and the abstract when summarising papers. I think you might be able to combat this by writing defensively - in our case we might have avoided that by saying "more information about geographic diversity may be found in X and the supplementary information"

Re: GPT-fabricated scientific papers on Google Scholar

#76

Post by kgeist »

I wonder how many of the GPT-generated papers are actually made by people whose native language is not English and who want to improve their English. That would explain various "as of my last knowledge update" still left intact in the papers, if the authors don't fully understand what it means.

Re: GPT-fabricated scientific papers on Google Scholar

#77

Post by zaptrem »

bravura wrote: post #71
Der_Einzige wrote: post #47

Earlier quoted context omitted.

With NeurIPS 2024 reviews going on right now, I'm sure that a whole lot of these kind of reviews are being generated daily.

With ICLR paper deadline coming up, I guess it's worth wargaming how GPT4 would review my submission.

Better yet, generate some adversarial perturbations to the text (or an invisible prompt) to cause it to give you a perfect review!

Re: GPT-fabricated scientific papers on Google Scholar

#79

Post by jampekka »

Strilanc wrote: post #26

When I went to the APS March Meeting earlier this year, I talked with the editor of a scientific journal and asked them if they were worried about LLM generated papers. They said actually their main worry wasn't LLM-generated papers, it was LLM-generated reviews . LLMs are much better at plausibly summarizing content than they are at doing long sequences of reasoning, so they're much better at generating believable r…

LLMs reviewing LLM generated articles via LLM editors is more or less guaranteed to become a massive thing given the incentive structures/survival pressures of everyone involved.

Researchers get massive CVs, reviewers and editors get off easy, admins get to show great output numbers from their institutions, and of course the publishers continue making hand over fist.

It's a rather broken system.

Re: GPT-fabricated scientific papers on Google Scholar

#80

Post by tkgally »

For a paper that includes both a broad discussion of the scholarly issues raised by LLMs and wide-ranging policy recommendations, I wish the authors had taken a more nuanced approach to data collection than just searching for “as of my last knowledge update” and/or “I don’t have access to real-time data” and weeding out the false positives manually. LLMs can be used in scholarly writing in many ways that will not be caught with such a coarse sieve. Some are obviously illegitimate, such as having an LLM write an entire paper with fabricated data. But there are other ways that are not so clearly unacceptable.

For example, the authors’ statement that “[GPT’s] undeclared use—beyond proofreading—has potentially far-reaching implications for both science and society” suggests that, for them, using LLMs for “proofreading” is okay. But “proofreading” is understood in various ways. For some people, it would include only correcting spelling and grammatical mistakes. For others, especially for people who are not native speakers of English, it can also include changing the wording and even rewriting entire sentences and paragraphs to make the meaning clearer. To what extent can one use an LLM for such revision without declaring that one has done so?

Post reply on HN