Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-fabricated scientific papers on Google Scholar

misinforeview.hks.harvard.edu

Post reply on HN
Print view View original

61–70 of 107 posts

Re: GPT-fabricated scientific papers on Google Scholar

#61

Post by jll29 »

matusp wrote: post #30
Strilanc wrote: post #26

When I went to the APS March Meeting earlier this year, I talked with the editor of a scientific journal and asked them if they were worried about LLM generated papers. They said actually their main worry wasn't LLM-generated papers, it was LLM-generated reviews . LLMs are much better at plausibly summarizing content than they are at doing long sequences of reasoning, so they're much better at generating believable r…

We already got an LLM generated meta review that was very clearly just summarization of reviews. There were some pretty egregious cases of borderline hallucinated remarks. This was ACL Rolling Review, so basically the most prestigious NLP venue and the editors told us to suck it up. Very disappointing and I genuinely worry about the state of science and how this will affect people who rely on scientometric criteria.

Most conferences have been flooded with submissions, and ACL is no exception.

A consequence of that is that there are not sufficient numbers of reviewers available who are qualified to review these manuscripts.

Conference organizers might be keen to accept many or most who offer to volunteer, but clearly there is now a large pool of people that have never done this before, and were never taught how to do this. Add some time pressure, and people will try out some tool, just because it exists.

GPT-generated docs have a particular tone that you can detect if you've played a bit with ChatGPT and if you have a feel for language. Such reviews should be kicked out. I would be interested to view this review (anonymized if you like - by taking out bits that reveal too narrowly what it's about).

The "rolling" model of ARR is a pain, though, because instead of slaving for a month you feel like slaving (conducting scientific peer review free of charge = slave labor) all year round. Last month, I got contacted by a book editor to review a scientific book for $100. I told her I'm not going to read 350 pages, to write two pages worth of book review; to do this properly one would need two days, and I quoted my consulting day rate. On top of that, this email came in the vacation month of August. Of course, said person was never heard of again.

Re: GPT-fabricated scientific papers on Google Scholar

#62

Post by anigbrowl »

OutOfHere wrote: post #7

Just because ChatGPT was used to help write a paper doesn't in itself mean that the data or findings are fabricated.

You can probably find some quality stuff in your local landfill too, but I am personally unwilling to sift through garbage.

Re: GPT-fabricated scientific papers on Google Scholar

#63

Post by hakanderyal »

ceejayoz wrote: post #15
OutOfHere wrote: post #7

Just because ChatGPT was used to help write a paper doesn't in itself mean that the data or findings are fabricated.

Sure, but there are some... pretty egregious cases. https://mashable.com/article/ai-rat-penis-diagram-midjourney...

That’s the funniest piece of writing I’ve read in a longtime, thanks!

I wonder what they were thinking submitting the paper.

Re: GPT-fabricated scientific papers on Google Scholar

#64

Post by jerpint »

Lerc wrote: post #55
refibrillator wrote: post #23

Hmm there may be a bug in the authors’ python script that searches google scholar for the phrases "as of my last knowledge update" or "I don't have access to real-time data". You can see the code in appendix B. The bug happens if the ‘bib’ key doesn’t exist in the api response. That leads to the urls array having more rows than the paper_data array. So the columns could become mismatched in the final data frame. It s…

As a tangent to the paper topic itself, what should be the standard procedure for publishing data gathering code like this? Given that they don't specify which version of any libraries or APIs used and that updates occur over time, API's change etc. inevitably resulting in code rot. It will eventually be impossible to figure out exactly what this code did. With meticulous version records it should at least be possibl…

Using a colab with printed outputs could be a good option to at the very least hint to reproducing results independently

Re: GPT-fabricated scientific papers on Google Scholar

#65

Post by EnigmaFlare »

pcrh wrote: post #60

This kind of fabricated result is not a problem for practitioners in the relevant fields, who can easily distinguish between false and real work. If there are instances where the ability to make such distinctions is lost, it is most likely to be so because the content lacks novelty, i.e. it simply regurgitates known and established facts. In which case it is a pointless effort, even if it might inflate the supposed a…

Non-experts actually attempting to become informed (instead of just feeling like they're informed) can easily tell the difference too. The people being fooled are the ones who want to be fooled. They're looking for something to support their pre-existing belief. And for those people, they'll always find something they can convince themselves supports their belief, so I don't think it matters what false information is floating around.

It seems to be kind of a new thing for laymen to be reading scientific papers. 20 years ago, they just weren't accessible. You had to physically go to a local university library and work out how to use the arcane search tools, which wouldn't really find what you wanted anyway. And even then, you couldn't take it home and half the time you couldn't even photocopy it because you needed a student ID card to use the photocopier.

Re: GPT-fabricated scientific papers on Google Scholar

#66

Post by Eisenstein »

basch wrote: post #56
Strilanc wrote: post #26

When I went to the APS March Meeting earlier this year, I talked with the editor of a scientific journal and asked them if they were worried about LLM generated papers. They said actually their main worry wasn't LLM-generated papers, it was LLM-generated reviews . LLMs are much better at plausibly summarizing content than they are at doing long sequences of reasoning, so they're much better at generating believable r…

It might follow to say that current LLM;s arent trained to generate papers, BUT they also don't really need to reason. They just need to mimic the appearance of reason, follow the same pattern of progression. Ingesting enough of what amounts to executed templates will teach it to generate its own results as if output from the same template.

What is the difference between 'reasoning' and 'appearing to be reasoning' if the results are the same with the same input?

Re: GPT-fabricated scientific papers on Google Scholar

#67

Post by EnigmaFlare »

layer8 wrote: post #2

I appreciate that, appropriately, the article image is not AI-generated.

It's silly that there's a stigma attached to AI generated images in cases where it's perfectly reasonable to do. People seem to appreciate things more for the fact that they were created by spending time out of another human's life more than what it actually is.

Re: GPT-fabricated scientific papers on Google Scholar

#68

Post by Eisenstein »

jll29 wrote: post #59
kovezd wrote: post #40

Earlier quoted context omitted.

I can see how LLMs contribute to raise the standard in that field. For example, surveying related research. Also, maybe in the not too distant future, reproducing (some) of the results.

Writing consists of iterated re-writing (to me, anyways), i.e. better and better ways to express content 1. correctly, 2. clearly and 3. space-economically. By writing it down (yourself) you understand what claims each piece of related work discussed has made (and can realistically make - as there sometims are inflationary lists of claims in papers), and this helps you formulate your own claim as it relates to them (…

Does every researcher write summaries of related research themselves?

Re: GPT-fabricated scientific papers on Google Scholar

#69

Post by nis0s »

Eisenstein wrote: post #66
basch wrote: post #56

Earlier quoted context omitted.

It might follow to say that current LLM;s arent trained to generate papers, BUT they also don't really need to reason. They just need to mimic the appearance of reason, follow the same pattern of progression. Ingesting enough of what amounts to executed templates will teach it to generate its own results as if output from the same template.

What is the difference between 'reasoning' and 'appearing to be reasoning' if the results are the same with the same input?

From what I’ve seen, the results are not the same. In the latter scenario, there’s a risk of encountering a non sequitur all of a sudden, and the citations may be nonexistent. There’s also no guarantee that what you’re stating is factually correct when your logic is unbounded by reality.

Re: GPT-fabricated scientific papers on Google Scholar

#70

Post by nis0s »

kenjackson wrote: post #25
hodgesrm wrote: post #9

> Two main risks arise... First, the abundance of fabricated “studies” seeping into all areas of the research infrastructure... A second risk lies in the increased possibility that convincingly scientific-looking content was in fact deceitfully created with AI tools... A third risk: ChatGPT has no understanding of "truth" in the sense of facts reported by established, trusted sources. I'm doing a research project rel…

It sounds like your use of AI is one of the worst uses. Standard semantic search would be much better and appropriate.

If summarization and analysis isn’t the main use of AI, then what is?
Post reply on HN