Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI intensifies fight against ‘paper mills’ that churn out fake research

nature.com

Post reply on HN
Print view View original

71–80 of 186 posts

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#71

Post by JacobThreeThree »

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

The current scientific system has long been known to have serious problems of incorrect results and conflicts of interest. This article seems like an attempt to pin the crisis on an AI scapegoat.

From 2015, the Editor of The Lancet:

The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”. The Academy of Medical Sciences, Medical Research Council, and Biotechnology and Biological Sciences Research Council have now put their reputational weight behind an investigation into these questionable research practices. The apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data. Journal editors deserve their fair share of criticism too. We aid and abet the worst behaviours. Our acquiescence to the impact factor fuels an unhealthy competition to win a place in a select few journals. Our love of “significance” pollutes the literature with many a statistical fairy-tale. We reject important confirmations. Journals are not the only miscreants. Universities are in a perpetual struggle for money and talent, endpoints that foster reductive metrics, such as high-impact publication. National assessment procedures, such as the Research Excellence Framework, incentivise bad practices. And individual scientists, including their most senior leaders, do little to alter a research culture that occasionally veers close to misconduct.

https://www.thelancet.com/journals/lancet/article/PIIS0140-6...

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#73

Post by jvanderbot »

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

The unfortunate truth is that academic lineage matters. In a world with decreasing SNR from paper content, the SNR from finding the paper on a reliable researcher's homepage massively increases (because sadly, people are added to papers without their permission in an effort to boost their own signal).

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#74

Post by goodpoint »

noughtme wrote: post #25
EGreg wrote: post #12

Earlier quoted context omitted.

Guns don’t kill people. They have no agency. But they sure make it easier.

Illegal guns kill people. Something like 80-90% of guns involved in homicide are illegally obtained/possessed. This is relevant because like banning guns, “banning” AI shouldn’t be the focus. It seems like an AI detection arms race is inevitable.

Then how comes a 10 year old cannot buy a gun or drive a car?

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#75

Post by probably_wrong »

malikNF wrote: post #66
huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

I understand how journals made sense during a time when distribution was a hard problem. Now with the internet I don’t get the reason for them to exist in the form it does in 2023.

A journal is a way of ensuring that the paper you are about to read meets a minimum standard of quality. My anecdote: the second worst paper I've ever read was an ArXiv paper, presented by a young researcher who had no idea of how bad this paper was, Had I not teared it down (which was a very unpleasant experience for everyone involved) it would have been further propagated by other young researchers. And the worst paper? Submitted to a conference, rejected outright during review, and I still wonder if someone was trying to sneak a machine-generated article for fun.

No one has the time to keep up with all the research coming out, not to mention all the fake research published in bad faith. Life is not restricted to Elsevier and ArXiv, and there are sane alternatives out there.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#76

Post by noobermin »

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

No offense, but astronomy is ripe for being rife with this sort of thing because the validity of astronomical research doesn't matter for the "real world" and can't really easily be tested. I feel like it could be better, but since you guys do not have any external pressures beyond yourselves (as academics), you can still publish however you feel like.

A lot of the more applied fields cannot survive scrutiny. Either you have to produce something that leads to a product or something a PM will scrutinize (especially if you work for a national lab or the like, or for pharma, and so on). There, a lot of published research might be bogus, but it is nowhere near the majority.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#77

Post by noobermin »

CobrastanJorji wrote: post #38
lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

Imagine a world where all the raw data sits in a public repository next to a script to process the data which was reviewed as part of the publication process, which would accompany the paper and could be quickly and easily replicated with new data. Anyone could produce the graphs shown in the paper simply by downloading both and running them together. What a wonderful thing that would be. And you could imagine the go…

I hate to be sincere, but the reality is the data is our product. If we open source our data too much, we won't have anything left to publish as others who would have to expend nowhere near the resources we must do to produce it can scrape it and publish it. (that literally is how "AI" of the current hype bubble works today, lol, why would I want that online?)

The incentives are definitely bad, and that's where the actual fix should be.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#78

Post by jrumbut »

simonw wrote: post #65
CobrastanJorji wrote: post #38

Earlier quoted context omitted.

Imagine a world where all the raw data sits in a public repository next to a script to process the data which was reviewed as part of the publication process, which would accompany the paper and could be quickly and easily replicated with new data. Anyone could produce the graphs shown in the paper simply by downloading both and running them together. What a wonderful thing that would be. And you could imagine the go…

Funding the storing and serving of all of that data doesn't sound like a difficult problem to me. That has gotten SO cheap over the past couple of decades. There are plenty of well funded institutions that can support that kind of resource.

You'd be surprised. One headache there is what did you tell study participants you would do with the data?

Did you say you'd keep it forever? Did you say 5 years? Who's in charge of making sure that this centralized repository isn't inappropriately holding and distributing data?

Funding organizations also have different requirements.

Then a script is only useful if paired with a set of libraries of a particular version, a specific compiler/interpreter, an OS, also there may be specialized hardware involved. Some of the languages used in science like SAS, Stata, SPSS, and Matlab, etc aren't free and open source so you can't always just bundle it.

And even data storage isn't trivial. For a recent small conference abstract I processed ~150GB of data. Hundreds (thousands? Tens of thousands?) of other papers have looked at that same data. You would really want some way to deduplicate that storage, but that introduces some additional complexity.

I do like this vision but I think it would be a major undertaking that would require a lot of well funded institutions coming together rather than any one in particular doing it on their own.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#79

Post by noobermin »

huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

Let's be real, there are a lot of papers in AI right now because it's been hot, not just recently but for years. Still, no one is reading random papers from insert-unknown-group-here. People read papers from already established scientists who have pedigree. And, how did they become established? Because they actually published before arxiv.

In that sense, peer review should be an equalizer, because the only way to be read and cited then is to have the holy academic lineage, which is what would happen if we abolished peer-review as it exists in the current academic word we're in.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#80

Post by naillo »

waboremo wrote: post #11
naillo wrote: post #6

The problem is the people doing this. The AI doesn't have agency, it's just an autocomplete tool.

Right so what solutions do you have here to prevent people from using the autocomplete tool for harm?

Punish the people using it, disallow and shame people who are found to use it. The usual stuff when you want people to stop doing things. (Not legally ban the whole population from making use of it even in cases where it's productive.)
Post reply on HN