Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI intensifies fight against ‘paper mills’ that churn out fake research

nature.com

Post reply on HN
Print view View original

51–60 of 186 posts

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#51

Post by CobrastanJorji »

huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

No, science definitely goes worse when there are more bad papers.

There's a popular argument about freedom of speech that comes down to "the solution to bad speech is more good speech, not limiting bad speech," but that only works (assuming it's true at all) when there's some way for a listener to discern truth with some research or effort.

> "Good papers are good because they have good arguments"

No they are not. Good papers are good when they collected data correctly and then presented the results fairly. Arguments about what that data means hardly enters into it. You can't argue whether the data is correct or not without gathering it yourself, and you can't gather it yourself.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#52

Post by eldaisfish »

huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

This line of thinking is dangerous and ignores very real problems as the other comments point out.

Instead of targeting peer-review, i recommend approaching this problem as one of incentives. Under the current system, what is the incentive for a reviewer to read and critique a paper? None. If they were paid in cash, there is a clear incentive. The publishers do not want this overheard and the associated legal requirements so they seek out "volunteers" and "compensate" them with rubbish like book copies and "reputation". The fault is with publishers, not the peer-review process.

At the moment, the current model allows parasites like Elsevier to derive the maximum benefit while not paying most folks involved in creating the actual value - the reviewers, editors and authors.

Citizen journalism tried this, failed and is now hollowing out established journalism. We need proper funding and proper incentives for journalism just as we need proper incentives for academic research.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#53

Post by SpaceBuddha »

bagels wrote: post #22
ushtaritk421 wrote: post #16

Feels like AI makes the fight against paper mills easier. A group could submit AI-generated papers to different places and create a public index based on how successful they are at getting their nonsense papers accepted.

https://en.wikipedia.org/wiki/Sokal_affair

"Later, after Sokal revealed the hoax in Lingua Franca, Social Text's editors wrote that they had requested editorial changes that Sokal refused to make, and had had concerns about the quality of the writing: 'We requested him (a) to excise a good deal of the philosophical speculation and (b) to excise most of his footnotes.' Still, despite calling Sokal a 'difficult, uncooperative author", and noting that such writers were 'well known to journal editors', based on Sokal's credentials Social Text published the article in the May 1996 Spring/Summer 'Science Wars' issue"

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#54

Post by joncrocks »

noughtme wrote: post #25
EGreg wrote: post #12

Earlier quoted context omitted.

Guns don’t kill people. They have no agency. But they sure make it easier.

Illegal guns kill people. Something like 80-90% of guns involved in homicide are illegally obtained/possessed. This is relevant because like banning guns, “banning” AI shouldn’t be the focus. It seems like an AI detection arms race is inevitable.

Apologies for taking the bait.

If the number of legal firearms were reduced then would that not reduce the pool from which the illegal firearms can be obtained?

Agreed it doesn't completely solve the issue, but could make it significantly less likely that one would encounter a firearm in general.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#55

Post by cortesoft »

CobrastanJorji wrote: post #51
huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

No, science definitely goes worse when there are more bad papers. There's a popular argument about freedom of speech that comes down to "the solution to bad speech is more good speech, not limiting bad speech," but that only works (assuming it's true at all) when there's some way for a listener to discern truth with some research or effort. > "Good papers are good because they have good arguments" No they are not. Go…

Yeah, the problem with that "more speech" argument is the imbalance... lies are way easier and faster to generate than truth, and can be crafted perfectly to play on what the audience wants to hear. The truth has to be the truth, so it can't be crafted in the same way.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#56

Post by stale2002 »

CobrastanJorji wrote: post #51
huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

No, science definitely goes worse when there are more bad papers. There's a popular argument about freedom of speech that comes down to "the solution to bad speech is more good speech, not limiting bad speech," but that only works (assuming it's true at all) when there's some way for a listener to discern truth with some research or effort. > "Good papers are good because they have good arguments" No they are not. Go…

> You can't argue whether the data is correct or not without gathering it yourself, and you can't gather it yourself.

You actually can argue about the data being "correct".

The way to do is, is studies need to be much more radically open with their raw data sources.

In AI, that would mean actually releasing the code you used, so people can try it themselves.

Or, in more sociology stuff, where you are doing, I don't know, interviews with humans, you could release the physical videos of your interviews.

And then for science stuff, show pictures/vidoes of the science you are doing.

I'm sure there would still be way to hack this. But significant more code and data transparency would do a huge amount for allowing other people to replicate or verify your work.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#57

Post by omginternets »

Prediction: as a result of fake research and it’s ilk (fake credentials, fake trade-journal pubs, fake reviews, etc.), we will observe a renewed interest in referrals and meatspace networking. It will become correspondingly difficult for outsiders to enter professional circles.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#58

Post by huijzer »

SantalBlush wrote: post #49
webnrrd2k wrote: post #32

Earlier quoted context omitted.

Re: "Science does not get worse when there are more bad papers"... I don't think that this is true at all. Weeding through bad papers is, at a minimum, an opportunity cost, as is a good paper built on top of a bad one. Also, there is a societal cost in that bad research can get picked up and believed by people, like the anti-vax crowd. Or, bad research can be used to push an agenda, like anti-climate change.

You are correct. Many people want research to work like a social media site where everyone can contribute, believing this is an egalitarian--and therefore better--solution. In reality, it would slow research to a crawl, as noise and conflicted interests dominate the conversation and drown out more rigorous or valuable information.

> noise and conflicted interests dominate the conversation and drown out more rigorous or valuable information.

That's actually a great point. Who says this isn't already the case? There isn't much evidence to suggest that things have gotten better after the Why Most Published Research Findings Are False paper in 2005. I think people should be extremely skeptical of anything they read, irregardless of a peer review stamp.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#59

Post by jxramos »

CobrastanJorji wrote: post #38
lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

Imagine a world where all the raw data sits in a public repository next to a script to process the data which was reviewed as part of the publication process, which would accompany the paper and could be quickly and easily replicated with new data. Anyone could produce the graphs shown in the paper simply by downloading both and running them together. What a wonderful thing that would be. And you could imagine the go…

seems attainable, doesn't sound far fetched technically. Maybe get a few universities to start the trend.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#60

Post by godelski »

There's really only one solution to this. Luckily it is the same solution to a lot of other things. Unluckily it is the same solution that no one has sought to implement for decades. Reviewing takes a lot of nuance, care, and time. You're not going to get this when all the incentives for the reviewers are currently to reject works. You can only get a group to work on ethics alone when that group is small and accountable. Incentivize the reviewers to be high quality and worthwhile work. Incentivize the chairs to review the reviewers and force high quality reviews. An expert paying close attention to a work makes the papermill's job exponentially more difficult. It also just makes the review process actually useful in the first place, as it ensures authors get actual feedback.
Post reply on HN