Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-fabricated scientific papers on Google Scholar

misinforeview.hks.harvard.edu

Post reply on HN
Print view View original

31–40 of 107 posts

Re: GPT-fabricated scientific papers on Google Scholar

#31

Post by tbrownaw »

Barrin92 wrote: post #22

Honestly what we need to do is establish much stronger credentialing schemes. The "only a good guy with an AI can stop a bad guy with an AI" approach of trying to filter out bad content is just a hopeless arms race and unproductive. In a sense we need to go back two steps and websites need to be much stronger curators of knowledge again, and we need some reliable ways to sign and attribute real authorship to publicat…

Is this not already a thing? You can look up purported papers by DOI, and whatever journal it came from supposedly had it reviewed and should know who sent it to them.

(And if that doesn't work, how is what you're suggesting meaningfully different?)

Re: GPT-fabricated scientific papers on Google Scholar

#32

Post by CuriouslyC »

riedel wrote: post #17
OutOfHere wrote: post #7

Just because ChatGPT was used to help write a paper doesn't in itself mean that the data or findings are fabricated.

True. I am seeing chatgpt used by my colleagues (mostly no English native speakers) day to day and it mostly improves their writing (except for those wotfd that pop up a bit too often [0] like utilize [1]). So not all bad. I am also hearing that a lot of reviewers and readers use it though. So we are often joking that PhD students (in CS) nowadays only write bullet point from their research. Generate prose that is us…

Scientific writing is pretty bad usually so I'll count this as an improvement

Re: GPT-fabricated scientific papers on Google Scholar

#33

Post by reliabilityguy »

matusp wrote: post #30
Strilanc wrote: post #26

When I went to the APS March Meeting earlier this year, I talked with the editor of a scientific journal and asked them if they were worried about LLM generated papers. They said actually their main worry wasn't LLM-generated papers, it was LLM-generated reviews . LLMs are much better at plausibly summarizing content than they are at doing long sequences of reasoning, so they're much better at generating believable r…

We already got an LLM generated meta review that was very clearly just summarization of reviews. There were some pretty egregious cases of borderline hallucinated remarks. This was ACL Rolling Review, so basically the most prestigious NLP venue and the editors told us to suck it up. Very disappointing and I genuinely worry about the state of science and how this will affect people who rely on scientometric criteria.

Well, given that the only thing that matters for tenure reviews is the “service”, i.e., roughly a list of conferences the applicant reviewed/performed some sort of service at, this is barely a surprise.

Right now there is now incentive to do a high quality review unless the reviewer is motivated.

Re: GPT-fabricated scientific papers on Google Scholar

#35

Post by viraptor »

refibrillator wrote: post #23

Hmm there may be a bug in the authors’ python script that searches google scholar for the phrases "as of my last knowledge update" or "I don't have access to real-time data". You can see the code in appendix B. The bug happens if the ‘bib’ key doesn’t exist in the api response. That leads to the urls array having more rows than the paper_data array. So the columns could become mismatched in the final data frame. It s…

That sounds important enough to contact the authors. Best case, they fixed it up manually; worst case, lots of papers are publicly accused of being made up and the whole farming/fish-focused summary they produced is completely wrong.

Re: GPT-fabricated scientific papers on Google Scholar

#36

Post by nextaccountic »

matusp wrote: post #30
Strilanc wrote: post #26

When I went to the APS March Meeting earlier this year, I talked with the editor of a scientific journal and asked them if they were worried about LLM generated papers. They said actually their main worry wasn't LLM-generated papers, it was LLM-generated reviews . LLMs are much better at plausibly summarizing content than they are at doing long sequences of reasoning, so they're much better at generating believable r…

We already got an LLM generated meta review that was very clearly just summarization of reviews. There were some pretty egregious cases of borderline hallucinated remarks. This was ACL Rolling Review, so basically the most prestigious NLP venue and the editors told us to suck it up. Very disappointing and I genuinely worry about the state of science and how this will affect people who rely on scientometric criteria.

Could you share it publicly or would you face adverse consequences?

If you can please publish it and maybe post here on HN or reddit.

Re: GPT-fabricated scientific papers on Google Scholar

#37

Post by viraptor »

hodgesrm wrote: post #9

> Two main risks arise... First, the abundance of fabricated “studies” seeping into all areas of the research infrastructure... A second risk lies in the increased possibility that convincingly scientific-looking content was in fact deceitfully created with AI tools... A third risk: ChatGPT has no understanding of "truth" in the sense of facts reported by established, trusted sources. I'm doing a research project rel…

> tried using ChatGPT to search for original sources

That's a bad idea, do not do that. Regardless of the the knowledge contained in ChatGPT, it's a completely wrong tool/tech - like using a jackhammer as a screwdriver. If your want original sources, then services like https://perplexity.ai can do it. It's not even an issue with ChatGPT as such, it was never intended for that - that's why they're trying to create search as well https://openai.com/index/searchgpt-prototype/

Re: GPT-fabricated scientific papers on Google Scholar

#38

Post by vunderba »

GaggiX wrote: post #8
judge2020 wrote: post #3

Earlier quoted context omitted.

I was able to get pretty close with chatgpt: https://rr.judge.sh/Commabutterfly/76b34e/nmwguWGt8pIe.jpg > create a picture of scrabble pieces strewn on a table, with a closeup of a line of scrabble letters spelling "CHATGPT" on top of them. photographic, realistic quality, maintain realism and believability

You can get much better results with Ideogram 2 (also free): https://ideogram.ai/assets/image/lossless/response/vF81gKjHS... https://ideogram.ai/assets/image/lossless/response/EcRpDLumS... Almost the same prompt.

You can also get reasonably close with an open model that you can run locally (flux dev).

https://replicate.com/p/xm41nvz05drm00chsywb6am7f0

https://replicate.com/p/kdw8bnkj39rm40chsyzbyg5e04

But of course anyone who has even a passing familiarity with scrabble is going to be able to tell that something's off.

Re: GPT-fabricated scientific papers on Google Scholar

#39

Post by viraptor »

jeremynixon wrote: post #6

There is article shows no evidence of fabrication, fraud or misinformation, while making accusations of all of them. All it shows is that ChatGPT was used, which is wildly escalated into "evidence manipulation" (ironically without evidence). Much more work is needed to show that this means anything.

If the result was not read even to check for obvious boilerplate GPT markers, then we can't expect anything else in them was. That means anything else, numbers, interpretation, conclusion was potentially never checked.

The authors use fraud in a specific sense here: "using ChatGPT fraudulently or undeclared" where they proved that the produced text was included without proper review. They also never accused those papers of misinformation, so they don't need to show evidence of that.

Re: GPT-fabricated scientific papers on Google Scholar

#40

Post by kovezd »

Strilanc wrote: post #26

When I went to the APS March Meeting earlier this year, I talked with the editor of a scientific journal and asked them if they were worried about LLM generated papers. They said actually their main worry wasn't LLM-generated papers, it was LLM-generated reviews . LLMs are much better at plausibly summarizing content than they are at doing long sequences of reasoning, so they're much better at generating believable r…

I can see how LLMs contribute to raise the standard in that field. For example, surveying related research. Also, maybe in the not too distant future, reproducing (some) of the results.
Post reply on HN