AI intensifies fight against ‘paper mills’ that churn out fake research • Hacker News Board

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#181

Post by lacker » Thu, Jun 01, 2023, 8:23 PM UTC

simonw wrote: post #65

CobrastanJorji wrote: post #38

Earlier quoted context omitted.

Imagine a world where all the raw data sits in a public repository next to a script to process the data which was reviewed as part of the publication process, which would accompany the paper and could be quickly and easily replicated with new data. Anyone could produce the graphs shown in the paper simply by downloading both and running them together. What a wonderful thing that would be. And you could imagine the go…

Funding the storing and serving of all of that data doesn't sound like a difficult problem to me. That has gotten SO cheap over the past couple of decades. There are plenty of well funded institutions that can support that kind of resource.

I can only really speak to astronomy, but the biggest problem is generally not hosting the raw data. The problem is maintaining the process of analyzing the data.

When you are doing science, you often do not just have a single standardized data format. It's not like taking a picture with a camera, where the jpg format is a standard and the metadata is a standard. If you are, say, storing data from a radio telescope, the metadata is more like taking a snapshots of a production database. Over time you might track additional data, like how much the telescope slewed recently, how much radio interference was nearby according to some new algorithm, etc etc.

Your data formats are constantly changing. So your analysis scripts are constantly changing as well. But this sort of thing doesn't just maintain itself. You very often need to add code like, how do you handle versions of the data where column X is missing. A research project might spend a year gathering data, and change the data format a tiny bit ten times over that year. If you publish something a few times a year, there eventually are a huge number of data versions and script versions that old publications rely on.

It isn't impossible to maintain data like this. You can have code that regularly runs integration tests and reruns past analyses. But most research doesn't operate to this level of "software engineering quality". One-off Jupyter notebooks, code that the developer only got working on their local machine, and so on.

I think we could do better, but it would involve hiring more software engineers and building engineering teams to support scientific research. It is not as simple as allocating more budget towards hosting large files.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#182

Post by ImPostingOnHN » Fri, Jun 02, 2023, 11:56 AM UTC

AnthonyMouse wrote: post #101

webnrrd2k wrote: post #32

Earlier quoted context omitted.

Re: "Science does not get worse when there are more bad papers"... I don't think that this is true at all. Weeding through bad papers is, at a minimum, an opportunity cost, as is a good paper built on top of a bad one. Also, there is a societal cost in that bad research can get picked up and believed by people, like the anti-vax crowd. Or, bad research can be used to push an agenda, like anti-climate change.

The solution to this is reputation. The flaw is that we're using the reputation of for-profit journals rather than the research institutions. You shouldn't have confidence in a paper because it was published in Nature, you should have confidence in it because it was published by Harvard or the University of California or Google Research, who puts the name of their institution on it and thereby stakes their reputation…

A research institution can produce bad researchers (Harvard, UC, and Google included), so we can't go with this suggestion either, instead we'd need reputation for each individual researcher in the world.

how many such reputations can you keep track of and gauge trust along?

and how is reputation gauged if peers don't review work?

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#183

Post by ImPostingOnHN » Fri, Jun 02, 2023, 12:26 PM UTC

mschuster91 wrote: post #91

webnrrd2k wrote: post #32

Earlier quoted context omitted.

Re: "Science does not get worse when there are more bad papers"... I don't think that this is true at all. Weeding through bad papers is, at a minimum, an opportunity cost, as is a good paper built on top of a bad one. Also, there is a societal cost in that bad research can get picked up and believed by people, like the anti-vax crowd. Or, bad research can be used to push an agenda, like anti-climate change.

> Also, there is a societal cost in that bad research can get picked up and believed by people, like the anti-vax crowd. Or, bad research can be used to push an agenda, like anti-climate change. Responsible media would act as a gatekeeper. The problem is, most media utterly gutted scientific journalists for more profit, so they completely lack the basis to evaluate and supply context on research. Others, particularly…

the gatekeeper job belongs to the publication, rather than people reporting on what the publication chose to publish, especially when the journalists reporting on it aren't experts in the field (which we obviously can't expect them to be for everything they report on)

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#184

Post by AnthonyMouse » Fri, Jun 02, 2023, 9:49 PM UTC

ImPostingOnHN wrote: post #182

AnthonyMouse wrote: post #101

Earlier quoted context omitted.

The solution to this is reputation. The flaw is that we're using the reputation of for-profit journals rather than the research institutions. You shouldn't have confidence in a paper because it was published in Nature, you should have confidence in it because it was published by Harvard or the University of California or Google Research, who puts the name of their institution on it and thereby stakes their reputation…

A research institution can produce bad researchers (Harvard, UC, and Google included), so we can't go with this suggestion either, instead we'd need reputation for each individual researcher in the world. how many such reputations can you keep track of and gauge trust along? and how is reputation gauged if peers don't review work?

> A research institution can produce bad researchers (Harvard, UC, and Google included), so we can't go with this suggestion either

Why not? The same is true of an individual researcher. But if they do, it damages their reputation, so they have an incentive to not. The same as the institution.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#185

Post by ImPostingOnHN » Sat, Jun 03, 2023, 2:59 PM UTC

AnthonyMouse wrote: post #184

ImPostingOnHN wrote: post #182

Earlier quoted context omitted.

A research institution can produce bad researchers (Harvard, UC, and Google included), so we can't go with this suggestion either, instead we'd need reputation for each individual researcher in the world. how many such reputations can you keep track of and gauge trust along? and how is reputation gauged if peers don't review work?

> A research institution can produce bad researchers (Harvard, UC, and Google included), so we can't go with this suggestion either Why not? The same is true of an individual researcher. But if they do, it damages their reputation, so they have an incentive to not. The same as the institution.

> Why not?

because it can produce bad researchers, and thus cannot be trusted to indicate good vs. bad researchers

thus, it is a bad proxy for trust of individual researchers

>The same is true of an individual researcher

individual researchers are not research institutions, so the same literally cannot be true: one employs the other, the inverse is not true

> if they do, it damages their reputation, so they have an incentive to not

empirically speaking, they objectively do, and their reputation is not damaged, and thus above proposition about reputation and incentives does not appear to be true enough to stop it from happening

recall the topic is how to gauge individual researcher reputation in the first place. Either we do it on an individual basis or a group/heuristic basis, and of the latter, research publications are a better proxy than what school one went to, but the former is better than both

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#186

Post by RosanaAnaDana » Tue, Jun 06, 2023, 5:42 PM UTC

mike_hearn wrote: post #132

RosanaAnaDana wrote: post #127

Earlier quoted context omitted.

I think if the data is valid, open access, and reproducible, it shouldn't matter if the paper was written by AI.

Unfortunately this assumes that the only thing which can go wrong is lack of reproducibility. Not so. I read a lot of public health papers during COVID and a staggering quantity (IMHO nearly all of them) should not have been published; many of them would have been reproducible despite that. Other things that can and do regularly appear in reproducible, peer reviewed papers: • Nonsensical methodologies • Logical falla…

I think you'll be demonstrated to be wrong in short order over a period of the next 1-2 years.

My called shot: We'll have fully automated scientific research with useful and valuable conclusions and little to no human intervention. I think it will take some time, but can tell you right now its on its way to happening.