Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI intensifies fight against ‘paper mills’ that churn out fake research

nature.com

Post reply on HN
Print view View original

141–150 of 186 posts

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#141

Post by jltsiren »

mohave529 wrote: post #137

From my perspective as an outsider, I have always been amazed by the use of papers in academic research as a means of communicating findings to the wider world. I find it problematic that these papers are often formatted in a way that makes them highly unreadable, with two columns and compressed text. In my opinion, adopting more modern methods of publishing research could greatly enhance the overall quality of resea…

Whatever you think academic research is, it's not always like that, and it's usually easy to find counterexamples. That's why the attempts to standardize the processes usually fail.

There is not necessarily any data behind the paper. Even if there is data, the conclusions may not be about the data. Even if the conclusions are about the data, the paper may not use quantitative methods. Even if it uses quantitative methods, confidence intervals may not make sense. Even if confidence intervals do make sense, adding new data might not. And so on.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#142

Post by Balgair »

For 'lay' people here on HN, I just want to make a quick point:

Peer-review does not mean that the reviewer is re-doing experiments or re-running analyses. They are only reviewing to see if the paper merits inclusion in the journal. Often this means telling the authors to do more experiments or check other things. But, to be clear, the peer-reviewer does not re-do things and check if they are 'right'

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#143

Post by SantalBlush »

mike_hearn wrote: post #140
SantalBlush wrote: post #138

Earlier quoted context omitted.

This is a great example of what I'm talking about. P-values of 0.049 is not a "tell"; that is an HN meme. You can have p-values above 5 percent and the paper can still convey novel and valid research, depending on the nature of the study. The 5-percent value is a common cutoff, but it is a largely arbitrary one. Sometimes there is justification for a higher one, and sometimes it should be lower. To gauge what sort of…

That's the point - it's an arbitrary cutoff, so the high frequency with which results cluster around that point is indicative of a problem. If a field accepts P=0.05 as significant it means 1 in 20 results can be false positives just by random chance. Now think about how many papers get published, and many of them report more than one thing. The right threshold should really be an order of magnitude lower. It's not O…

>If a field accepts P=0.05 as significant it means 1 in 20 results can be false positives just by random chance.

That is not the correct interpretation of a p-value. See the ASA's statement on p-values.[0]

>Researchers often wish to turn a p-value into a state- ment about the truth of a null hypothesis, or about the probability that random chance produced the observed data. The p-value is neither. It is a statement about data in relation to a specified hypothetical explanation, and is not a statement about the explanation itself.

Moreover, while it's fine to suspect that p-hacking took place with p-values just under 5 percent, it is merely a suspicion and nothing more. Throwing out p-values you don't like without further evidence is a different sort of violation of the methodology; it is like p-hacking in the other direction.

>The right threshold should really be an order of magnitude lower.

This is false. Again, see the ASA's statement.[0]

>Practices that reduce data analysis or scientific infer- ence to mechanical “bright-line” rules (such as “p In short, these things often require a strong statistical literacy to interpret, which most people complaining about p-hacking do not possess.

[0]https://amstat.tandfonline.com/doi/pdf/10.1080/00031305.2016...

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#144

Post by __MatrixMan__ »

The problem is letting this be a step that directs our attention in the first place:

> platform X published this so it must be good

It's a root-of-trust scheme, and those create high value targets which fail to corruption. Better would be:

> human Y cited this, it's about Z, and you've configured Y to be trusted in domain Z

Webs of trust require maintenance, which isn't convenient, but if roots of trust continue to degrade in trustworthiness, then that maintenance will eventually be a price worth paying.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#145

Post by JoshuaJB »

Balgair wrote: post #142

For 'lay' people here on HN, I just want to make a quick point: Peer-review does not mean that the reviewer is re-doing experiments or re-running analyses. They are only reviewing to see if the paper merits inclusion in the journal. Often this means telling the authors to do more experiments or check other things. But, to be clear, the peer-reviewer does not re-do things and check if they are 'right'

Although there is a type of peer review that includes redoing experiments and analysis: artifact evaluation. All of the top conferences in my field (real-time embedded systems) include this as an opt-in option, and papers get a special badge if they also pass artifact evaluation. I strongly believe that other fields in computer science would benefit by including and normalizing this process.

Besides the reproducibility benefits, artifact evaluation forces documentation of the experiments and process; I've found this enormously useful when on-boarding new students to an existing project.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#146

Post by gtop3 »

__MatrixMan__ wrote: post #144

The problem is letting this be a step that directs our attention in the first place: > platform X published this so it must be good It's a root-of-trust scheme, and those create high value targets which fail to corruption. Better would be: > human Y cited this, it's about Z, and you've configured Y to be trusted in domain Z Webs of trust require maintenance, which isn't convenient, but if roots of trust continue to d…

The height of trust rot is Science publishing Woo Suk Hwang's 2005 stem cell research. This was one of the top scientific journals publishing research that appeared to be a nobel-prize track line of research that would have been a breakthrough in medical treatment. Instead it tainted a line of research.

The research results were fraudulent, claiming a much higher success rate at generating a stem cell line than what was achieved. They lied about the number of stem cell lines generated, the number of oocytes used to generate the stem cell lines, and the number of donors the oocytes came from.

As if bad data wasn't enough, they lied both to their donors, lied about their donors, and miscredited authors.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#147

Post by gtop3 »

noobermin wrote: post #76
lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

No offense, but astronomy is ripe for being rife with this sort of thing because the validity of astronomical research doesn't matter for the "real world" and can't really easily be tested. I feel like it could be better, but since you guys do not have any external pressures beyond yourselves (as academics), you can still publish however you feel like. A lot of the more applied fields cannot survive scrutiny. Either…

Astronomy is fantastic for preventing this type of fraud.

The data is incredibly open. Anyone can download datasets from telescopes, or build their own. Papers built on invalid data can be refuted by simply pointing telescopes at the location the paper is lying about. There are very few possible discoveries or theories in an astronomy journal that would immediately impact the wider culture.

Compare astronomy to sociology, in which questionnaire results can be fabricated and are essentially unconformable without using other unconformable questionnaires. Compare astronomy to economics, which the articles can be used bolster political positions.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#148

Post by ftxbro »

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

> "The idea that peer review is enough to ensure that research is valid is perhaps not scaling well."

As others have said, this is not what peer review has ever been for, at all. It only checks for gross omissions and violations of form and syntax that are obvious to other scientists who are in adjacent fields (not even necessarily the same one). It's a relict from the times when publications were not target metrics. Peer reviewers never replicate the work. If you are a grad student tapped for peer review and you spend the time to replicate the work, you are ruining your career and your advisor will also be mad.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#149

Post by __MatrixMan__ »

gtop3 wrote: post #146
__MatrixMan__ wrote: post #144

The problem is letting this be a step that directs our attention in the first place: > platform X published this so it must be good It's a root-of-trust scheme, and those create high value targets which fail to corruption. Better would be: > human Y cited this, it's about Z, and you've configured Y to be trusted in domain Z Webs of trust require maintenance, which isn't convenient, but if roots of trust continue to d…

The height of trust rot is Science publishing Woo Suk Hwang's 2005 stem cell research. This was one of the top scientific journals publishing research that appeared to be a nobel-prize track line of research that would have been a breakthrough in medical treatment. Instead it tainted a line of research. The research results were fraudulent, claiming a much higher success rate at generating a stem cell line than what…

I wasn't aware of the this kind of thing in 2005. How do you think the trustworthiness of papers has been trending since then, generally speaking. It sounds like that was a bit of an outlier.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#150

Post by __MatrixMan__ »

webnrrd2k wrote: post #32
huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

Re: "Science does not get worse when there are more bad papers"... I don't think that this is true at all. Weeding through bad papers is, at a minimum, an opportunity cost, as is a good paper built on top of a bad one. Also, there is a societal cost in that bad research can get picked up and believed by people, like the anti-vax crowd. Or, bad research can be used to push an agenda, like anti-climate change.

> Weeding through bad papers

How does one come across these bad papers in the first place, such that they must be weeded through? Maybe we need a different method of curation such that they stay below the noise floor.

Post reply on HN