Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-fabricated scientific papers on Google Scholar

misinforeview.hks.harvard.edu

Post reply on HN
Print view View original

41–50 of 107 posts

Re: GPT-fabricated scientific papers on Google Scholar

#41

Post by hiddencost »

refibrillator wrote: post #23

Hmm there may be a bug in the authors’ python script that searches google scholar for the phrases "as of my last knowledge update" or "I don't have access to real-time data". You can see the code in appendix B. The bug happens if the ‘bib’ key doesn’t exist in the api response. That leads to the urls array having more rows than the paper_data array. So the columns could become mismatched in the final data frame. It s…

https://www.hb.se/en/research/research-portal/researchers/JU...

Contact info for the first author

Re: GPT-fabricated scientific papers on Google Scholar

#42

Post by HeatrayEnjoyer »

kenjackson wrote: post #25
hodgesrm wrote: post #9

> Two main risks arise... First, the abundance of fabricated “studies” seeping into all areas of the research infrastructure... A second risk lies in the increased possibility that convincingly scientific-looking content was in fact deceitfully created with AI tools... A third risk: ChatGPT has no understanding of "truth" in the sense of facts reported by established, trusted sources. I'm doing a research project rel…

It sounds like your use of AI is one of the worst uses. Standard semantic search would be much better and appropriate.

How do you run a semantic search

Re: GPT-fabricated scientific papers on Google Scholar

#43

Post by GaggiX »

vunderba wrote: post #38
GaggiX wrote: post #8

Earlier quoted context omitted.

You can get much better results with Ideogram 2 (also free): https://ideogram.ai/assets/image/lossless/response/vF81gKjHS... https://ideogram.ai/assets/image/lossless/response/EcRpDLumS... Almost the same prompt.

You can also get reasonably close with an open model that you can run locally (flux dev). https://replicate.com/p/xm41nvz05drm00chsywb6am7f0 https://replicate.com/p/kdw8bnkj39rm40chsyzbyg5e04 But of course anyone who has even a passing familiarity with scrabble is going to be able to tell that something's off.

The biggest problem with the default Flux model is that it generates images with that strong AI look, probably caused by the distillation of the CFG. You should try some LoRAs for this, and also prompt the model to generate the rack that holds the letters.

Re: GPT-fabricated scientific papers on Google Scholar

#44

Post by viraptor »

gerdesj wrote: post #11

Colour me surprised. An IT related search will generally end up with loads of returns that lead to AI generated wankery. For example, suppose you wish to back up switch configs or dump a file or whatever and tftp is so easy and simple to setup. You'll tear it down later or firewall it or whatever. So a quick search "linux tftp serevr" gets you to say: https://thelinuxcode.com/install_tftp_server_ubuntu/ All good unti…

That's not an AI hallucination. The content comes from Ubuntu community wiki https://help.ubuntu.com/community/TFTP - it was written in 2015. And at least in Debian, tftpd-hpa man page lists --create as valid https://manpages.debian.org/testing/tftpd-hpa/tftpd.8.en.htm...

Seems valid upstream too https://github.com/Distrotech/tftp-hpa/blob/5e95f248e8435eb3...

Re: GPT-fabricated scientific papers on Google Scholar

#45

Post by cratermoon »

OutOfHere wrote: post #7

Just because ChatGPT was used to help write a paper doesn't in itself mean that the data or findings are fabricated.

The problem is not that a paper has fabricated content generated by ChatGPT, the problem is that there are many papers and they are polluting scholarship to the point that the base of evidence used in policy-making could be poisoned to the point of uselessness.

Re: GPT-fabricated scientific papers on Google Scholar

#46

Post by mkl »

gerdesj wrote: post #11

Colour me surprised. An IT related search will generally end up with loads of returns that lead to AI generated wankery. For example, suppose you wish to back up switch configs or dump a file or whatever and tftp is so easy and simple to setup. You'll tear it down later or firewall it or whatever. So a quick search "linux tftp serevr" gets you to say: https://thelinuxcode.com/install_tftp_server_ubuntu/ All good unti…

It says to put the --create option in /etc/default/tftpd-hpa. tftpd-hpa does support --create (at least on Ubuntu). The client program tftp-hpa (no d) doesn't support --create, but that's not what the instructions are talking about.

Re: GPT-fabricated scientific papers on Google Scholar

#47

Post by Der_Einzige »

matusp wrote: post #30
Strilanc wrote: post #26

When I went to the APS March Meeting earlier this year, I talked with the editor of a scientific journal and asked them if they were worried about LLM generated papers. They said actually their main worry wasn't LLM-generated papers, it was LLM-generated reviews . LLMs are much better at plausibly summarizing content than they are at doing long sequences of reasoning, so they're much better at generating believable r…

We already got an LLM generated meta review that was very clearly just summarization of reviews. There were some pretty egregious cases of borderline hallucinated remarks. This was ACL Rolling Review, so basically the most prestigious NLP venue and the editors told us to suck it up. Very disappointing and I genuinely worry about the state of science and how this will affect people who rely on scientometric criteria.

With NeurIPS 2024 reviews going on right now, I'm sure that a whole lot of these kind of reviews are being generated daily.

Re: GPT-fabricated scientific papers on Google Scholar

#48

Post by userbinator »

matusp wrote: post #30
Strilanc wrote: post #26

When I went to the APS March Meeting earlier this year, I talked with the editor of a scientific journal and asked them if they were worried about LLM generated papers. They said actually their main worry wasn't LLM-generated papers, it was LLM-generated reviews . LLMs are much better at plausibly summarizing content than they are at doing long sequences of reasoning, so they're much better at generating believable r…

We already got an LLM generated meta review that was very clearly just summarization of reviews. There were some pretty egregious cases of borderline hallucinated remarks. This was ACL Rolling Review, so basically the most prestigious NLP venue and the editors told us to suck it up. Very disappointing and I genuinely worry about the state of science and how this will affect people who rely on scientometric criteria.

so basically the most prestigious NLP venue

I see "dogfooding" has now been taken to its natural conclusion.

Re: GPT-fabricated scientific papers on Google Scholar

#49

Post by Barrin92 »

tbrownaw wrote: post #31
Barrin92 wrote: post #22

Honestly what we need to do is establish much stronger credentialing schemes. The "only a good guy with an AI can stop a bad guy with an AI" approach of trying to filter out bad content is just a hopeless arms race and unproductive. In a sense we need to go back two steps and websites need to be much stronger curators of knowledge again, and we need some reliable ways to sign and attribute real authorship to publicat…

Is this not already a thing? You can look up purported papers by DOI, and whatever journal it came from supposedly had it reviewed and should know who sent it to them. (And if that doesn't work, how is what you're suggesting meaningfully different?)

It's not at all a thing. Here's a recent study looking at citation fraud on Google Scholar including professional citation boosting services including with fake identities. It's widespread practice. https://arxiv.org/abs/2402.04607

Having a machine verifiable, cryptographic identity system that renders these kinds of things transparent, basically the equivalent of a ledger but instead of using it for get-rich schemes using it for identity would probably make verification enforceable.

Re: GPT-fabricated scientific papers on Google Scholar

#50

Post by [deleted] »

bongodongobob wrote: post #20
gerdesj wrote: post #11

Colour me surprised. An IT related search will generally end up with loads of returns that lead to AI generated wankery. For example, suppose you wish to back up switch configs or dump a file or whatever and tftp is so easy and simple to setup. You'll tear it down later or firewall it or whatever. So a quick search "linux tftp serevr" gets you to say: https://thelinuxcode.com/install_tftp_server_ubuntu/ All good unti…

It's funny you mention this because yesterday I had it write me a shell script to set up a TFTP server from scratch. I had it walk me through the process first, then said "ok now make that into a script." And it did and it works.

[deleted]
Post reply on HN