AI intensifies fight against ‘paper mills’ that churn out fake research • Hacker News Board

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#61

Post by huijzer » Wed, May 31, 2023, 4:38 PM UTC

CobrastanJorji wrote: post #51

huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

No, science definitely goes worse when there are more bad papers. There's a popular argument about freedom of speech that comes down to "the solution to bad speech is more good speech, not limiting bad speech," but that only works (assuming it's true at all) when there's some way for a listener to discern truth with some research or effort. > "Good papers are good because they have good arguments" No they are not. Go…

> There's a popular argument about freedom of speech that comes down to "the solution to bad speech is more good speech, not limiting bad speech," but that only works (assuming it's true at all) when there's some way for a listener to discern truth with some research or effort.

Yes exactly. If it is too hard for a group of scientists to figure out a flaw, then it is also too hard for two peer reviewers to figure out a flaw. Only time can detect all errors in science, assuming that nonconformist papers are allowed into the conversation.

> Good papers are good when they collected data correctly and then presented the results fairly.

This is only true for empirical papers. Good data is a part of a good argument in the case of empirical papers.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#62

Post by venv » Wed, May 31, 2023, 4:42 PM UTC

CobrastanJorji wrote: post #38

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

Imagine a world where all the raw data sits in a public repository next to a script to process the data which was reviewed as part of the publication process, which would accompany the paper and could be quickly and easily replicated with new data. Anyone could produce the graphs shown in the paper simply by downloading both and running them together. What a wonderful thing that would be. And you could imagine the go…

Even this would require somehow verifying the raw data. It's plausible a bad actor could "reverse engineer" their data from a pre-determined conclusion.

But yes, overall more openness is good. Still, the cost losing trust in society is very high (as you need to verify everything).

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#63

Post by godelski » Wed, May 31, 2023, 4:44 PM UTC

omginternets wrote: post #57

Prediction: as a result of fake research and it’s ilk (fake credentials, fake trade-journal pubs, fake reviews, etc.), we will observe a renewed interest in referrals and meatspace networking. It will become correspondingly difficult for outsiders to enter professional circles.

On the other hand, maybe we'll abandon peer review and return to a pre 1950's like style? Everyone in ML just uses arxiv because the field is moving so fast. All work is realistically peer reviewed once it is out there. We treat conferences (more important than journals) as very noisy signals. Unfortunately, we still use those as metrics for completing degrees or hiring people.

The way I would try to make this better is just to post to OpenReview rather than arxiv or integrate the two. That way discussions can be had on the works, in the open, and public.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#64

Post by reprociboi » Wed, May 31, 2023, 4:45 PM UTC

CobrastanJorji wrote: post #38

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

Imagine a world where all the raw data sits in a public repository next to a script to process the data which was reviewed as part of the publication process, which would accompany the paper and could be quickly and easily replicated with new data. Anyone could produce the graphs shown in the paper simply by downloading both and running them together. What a wonderful thing that would be. And you could imagine the go…

Some journals in fact implement this idea (e.g. having the raw data underlying each figure one click away). That is however not the crux of the reproducibility crisis; it would be great if it was just "I can't make an exact copy of Supplementary Figure 5D", but rather "I can't confirm that protein X causes dementia using orthogonal techniques". There is no easy code fix for that problem.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#65

Post by simonw » Wed, May 31, 2023, 4:45 PM UTC

CobrastanJorji wrote: post #38

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

Imagine a world where all the raw data sits in a public repository next to a script to process the data which was reviewed as part of the publication process, which would accompany the paper and could be quickly and easily replicated with new data. Anyone could produce the graphs shown in the paper simply by downloading both and running them together. What a wonderful thing that would be. And you could imagine the go…

Funding the storing and serving of all of that data doesn't sound like a difficult problem to me. That has gotten SO cheap over the past couple of decades. There are plenty of well funded institutions that can support that kind of resource.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#66

Post by malikNF » Wed, May 31, 2023, 4:45 PM UTC

huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

I understand how journals made sense during a time when distribution was a hard problem. Now with the internet I don’t get the reason for them to exist in the form it does in 2023.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#67

Post by justrealist » Wed, May 31, 2023, 4:46 PM UTC

jfengel wrote: post #36

noughtme wrote: post #25

Earlier quoted context omitted.

Illegal guns kill people. Something like 80-90% of guns involved in homicide are illegally obtained/possessed. This is relevant because like banning guns, “banning” AI shouldn’t be the focus. It seems like an AI detection arms race is inevitable.

Illegal guns start as legal guns. With both AI and guns you need to be willing to think about the problem at every step in the chain, not just at the end.

> Illegal guns start as legal guns.

To be pedantic, there are very few actually illegal guns. Mostly we are talking about crimes committed by felons who have committed prior crimes are no longer legally allowed access to firearms.

That same gun, if possessed by a non-felon, would be legal.

I don't know if this is or isn't important to the analogy, but I do think it applies.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#68

Post by venv » Wed, May 31, 2023, 4:47 PM UTC

samstave wrote: post #24

captainbland wrote: post #13

Earlier quoted context omitted.

This seems like a bleak dystopian fiction concept actually. AIs being in control of human experimentation on the basis of well meaning but poorly interpreted ethical guidelines (with, of course, fully hallucinated loopholes). Maybe add in that this has become one of the few sources of income for people as more and more jobs have been lost to the machines so many are compelled to participate.

" Take all the research done in [field] and compare the papers written by N authors, and look into their citations and compare the outcomes of what was published and find where the summation of the reads of their papers as shared citations and summarize the research where they agree, disagree and compare with international organizations and their papers coming from specifically the leads in this field from countries…

What you are describing, is not replication.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#69

Post by SantalBlush » Wed, May 31, 2023, 4:51 PM UTC

huijzer wrote: post #58

SantalBlush wrote: post #49

Earlier quoted context omitted.

You are correct. Many people want research to work like a social media site where everyone can contribute, believing this is an egalitarian--and therefore better--solution. In reality, it would slow research to a crawl, as noise and conflicted interests dominate the conversation and drown out more rigorous or valuable information.

> noise and conflicted interests dominate the conversation and drown out more rigorous or valuable information. That's actually a great point. Who says this isn't already the case? There isn't much evidence to suggest that things have gotten better after the Why Most Published Research Findings Are False paper in 2005. I think people should be extremely skeptical of anything they read, irregardless of a peer review s…

>I think people should be extremely skeptical of anything they read

This is a mistake that many HN readers make; they think that if one is equipped with some above-average level of intelligence, one can discern the validity of new research. But this is wrong. It usually takes many years of study before one can begin to clearly understand what is even being said, let alone whether it has any veracity.

People of above-average intelligence often resent this fact, because it suggests that their smart opinion isn't as valuable as the opinion of an expert, but that is simply a sad fact of life. If they haven't put in the work in that field, then they don't know what they're talking about in that field, and they are incapable of applying skepticism in that field correctly. This is precisely and unfortunately why we are forced to place our trust in experts.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#70

Post by venv » Wed, May 31, 2023, 4:51 PM UTC

huijzer wrote: post #58

SantalBlush wrote: post #49

Earlier quoted context omitted.

You are correct. Many people want research to work like a social media site where everyone can contribute, believing this is an egalitarian--and therefore better--solution. In reality, it would slow research to a crawl, as noise and conflicted interests dominate the conversation and drown out more rigorous or valuable information.

> noise and conflicted interests dominate the conversation and drown out more rigorous or valuable information. That's actually a great point. Who says this isn't already the case? There isn't much evidence to suggest that things have gotten better after the Why Most Published Research Findings Are False paper in 2005. I think people should be extremely skeptical of anything they read, irregardless of a peer review s…

Maybe, but it would still be better if the situation didn't detoriate further.