AI intensifies fight against ‘paper mills’ that churn out fake research • Hacker News Board

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#121

Post by tga_d » Wed, May 31, 2023, 7:26 PM UTC

JacobThreeThree wrote: post #71

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

The current scientific system has long been known to have serious problems of incorrect results and conflicts of interest. This article seems like an attempt to pin the crisis on an AI scapegoat. From 2015, the Editor of The Lancet: The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exp…

How do you see this as scapegoating? The headline specifically says "intensifies", the article very clearly positions AI-fabricated data as an extension of existing problems, and I don't see anything in the article downplaying those existing problems (the entire closing section is about how the summit was on issues broader than AI).

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#122

Post by matthewdgreen » Wed, May 31, 2023, 7:27 PM UTC

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

I love the idea. But do be aware that the high-quality software engineering like this will require has a cost: be prepared for basic science costs to scale appropriately (or for productivity to be reduced.) More to the point, not all science can be usefully verified by unit tests. Even machine-verified mathematical proofs are entirely dependent on the definitions being correct, and that requires expert human analysis.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#123

Post by ChainOfFools » Wed, May 31, 2023, 7:27 PM UTC

diognesofsinope wrote: post #92

noobermin wrote: post #77

Earlier quoted context omitted.

I hate to be sincere, but the reality is the data is our product. If we open source our data too much, we won't have anything left to publish as others who would have to expend nowhere near the resources we must do to produce it can scrape it and publish it. (that literally is how "AI" of the current hype bubble works today, lol, why would I want that online?) The incentives are definitely bad, and that's where the a…

If any level of government funds your institution you should have to releases the data. The code I write is not my code, it's the banks.

Most projects are funded by a patchwork of different sources, not all of which would agree to the same terms of release of the information, but all of which are required sufficiently fund its creation. Not to mention maintaining ongoing storage and accessibility. single source government grants for scientific research are the exception, not the rule.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#124

Post by AnthonyMouse » Wed, May 31, 2023, 7:30 PM UTC

notahacker wrote: post #120

AnthonyMouse wrote: post #101

Earlier quoted context omitted.

The solution to this is reputation. The flaw is that we're using the reputation of for-profit journals rather than the research institutions. You shouldn't have confidence in a paper because it was published in Nature, you should have confidence in it because it was published by Harvard or the University of California or Google Research, who puts the name of their institution on it and thereby stakes their reputation…

The research institutes are also for profit bodies, just much more of an closed shop because there's only so many people they can employ at any one time (and that's before we even get into tenure or the fact all you're really doing is shifting the peer review burden onto people that report into the person that wrote the paper). You can have papers published by Google Research or by nobody now: turns out academics lik…

> The research institutes are also for profit bodies

Their business model isn't to put the research behind a paywall, which is the relevant distinction.

> turns out academics like the journals' attempts to curate interesting stuff more than only ever reading stuff on their bookmark list or that's mailed to them in a desperate attempt to get some engagement.

This is what conferences are for, and then you have the conference organizers curating the talks but everyone gets the list of talks and links to all the freely available papers even if they can't afford to attend the conference.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#125

Post by ChainOfFools » Wed, May 31, 2023, 7:31 PM UTC

venv wrote: post #62

CobrastanJorji wrote: post #38

Earlier quoted context omitted.

Imagine a world where all the raw data sits in a public repository next to a script to process the data which was reviewed as part of the publication process, which would accompany the paper and could be quickly and easily replicated with new data. Anyone could produce the graphs shown in the paper simply by downloading both and running them together. What a wonderful thing that would be. And you could imagine the go…

Even this would require somehow verifying the raw data. It's plausible a bad actor could "reverse engineer" their data from a pre-determined conclusion. But yes, overall more openness is good. Still, the cost losing trust in society is very high (as you need to verify everything).

> It's plausible a bad actor could "reverse engineer" their data from a pre-determined conclusion.

I've already heard of someone planning a product ( initially targeted at lazy^H^H^H^Hbusy high schoolers and undergrads) that will use AI to reverse discover citations that fit a predetermined narrative in a research paper. Write whatever the hell you want, and the AI will do its best to backsolve a pile of citations that support your unsourced claims and arguments. The founder, and I use that term very generously, expects the academic community to net support this because it will boost citation counts for the vast majority of low citation, low visibility works.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#126

Post by [deleted] » Wed, May 31, 2023, 7:50 PM UTC

diognesofsinope wrote: post #92

noobermin wrote: post #77

Earlier quoted context omitted.

I hate to be sincere, but the reality is the data is our product. If we open source our data too much, we won't have anything left to publish as others who would have to expend nowhere near the resources we must do to produce it can scrape it and publish it. (that literally is how "AI" of the current hype bubble works today, lol, why would I want that online?) The incentives are definitely bad, and that's where the a…

If any level of government funds your institution you should have to releases the data. The code I write is not my code, it's the banks.

[deleted]

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#127

Post by RosanaAnaDana » Wed, May 31, 2023, 7:58 PM UTC

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

I think if the data is valid, open access, and reproducible, it shouldn't matter if the paper was written by AI.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#128

Post by godelski » Wed, May 31, 2023, 7:59 PM UTC

noobermin wrote: post #84

godelski wrote: post #60

There's really only one solution to this. Luckily it is the same solution to a lot of other things. Unluckily it is the same solution that no one has sought to implement for decades. Reviewing takes a lot of nuance, care, and time. You're not going to get this when all the incentives for the reviewers are currently to reject works. You can only get a group to work on ethics alone when that group is small and accounta…

1) You must therefore pay the reviewers. Probably a good starting point. 2) It is in fact, hard to publish things already in top journals especially if you're not established. Given the fact you also want the pool of reviewers to shrink, you're also making difficulty of publishing works even harder. Yes, the issue is paper mills, got it, but you're making it even harder for sincere scientists who aren't part of a mil…

1) I mostly agree. There are at least other rewards that can also be offered, such as conference discounts. There's also the token system (pay tokens to submit works, receive tokens for reviewing). But some incentive structure needs to happen, agreed.

> especially if you're not established

This is commonly stated but I don't think many internalize what it means. I don't think a meritocracy can exist -- this indicator supports that -- and shows that there's strong factors influencing acceptance beyond the quality of work.

2) I do think we need to be very careful about how we sell the prestige of publications/venues. As fields get more popular and our metrics rely more on them (publish or perish) then this not only encourages paper mills, but cheating in general. As two simple examples, look at how ML publishes works where papers use proprietary datasets/models (reveals authors' lab and thus violates ethics), or how it is status quo to tune hyperparameters on test data results (information leakage). You're are a huge disadvantage if you don't cheat. We have a similar problem in schools and this is why students cheat. If it is hard to catch and bad actors aren't punished (high risk because high false positive) then you actively reward cheaters. This needs to be a serious conversation and I don't think we are having this (this is several domains, even outside academia).

3) There's a coupling effect here though, that is pretty destructive and can have bigger social ramifications. As the noise in venue publication increases, the trust in the venue decreases. But I think it increases first, as people are metric chasing. But I suspect it'll be like a rubber band, and snap back hard. The larger ramification is social trust around science, where only the large venues matter and not the small unknown journals. We already have a growing distrust as the "just asking questions" anti-science strategy has been growing, and we need to be pretty careful and think beyond a local (spatially and temporally) window.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#129

Post by tga_d » Wed, May 31, 2023, 8:07 PM UTC

RosanaAnaDana wrote: post #127

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

I think if the data is valid, open access, and reproducible, it shouldn't matter if the paper was written by AI.

The article says this:

> Kahn says that, although there will undoubtedly be positive uses of AI to support researchers writing papers, it will still be necessary to distinguish between legitimate papers written with AI and those that have been completely fabricated.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#130

Post by omginternets » Wed, May 31, 2023, 8:12 PM UTC

joncrocks wrote: post #54

noughtme wrote: post #25

Earlier quoted context omitted.

Illegal guns kill people. Something like 80-90% of guns involved in homicide are illegally obtained/possessed. This is relevant because like banning guns, “banning” AI shouldn’t be the focus. It seems like an AI detection arms race is inevitable.

Apologies for taking the bait. If the number of legal firearms were reduced then would that not reduce the pool from which the illegal firearms can be obtained? Agreed it doesn't completely solve the issue, but could make it significantly less likely that one would encounter a firearm in general.

Likely, yes. One of the practical difficulties in the US, however, is that the pool has historically been replenished by the trafficking of guns. So yes, reducing the pool of available guns is prima facie sensible, but it takes a fair bit more than declaring guns illegal. Further complicating the issue is the fact that guns have a very long service life. I have an FB Vis from 1942 that still works flawlessly, despite having seen years of service in actual war. With ordinary care and the replacement of high-wear parts (the extractor, mostly, and maybe the barrel), I’d expect it to run at least another 100k rounds.

The proposition to reduce gun violence by reducing the number of guns in circulation has to account for the guns already in circulation, and for the propensity of black markets to introduce new guns into circulation. Otherwise, the only thing achieved is the disarming of the groups of people with legitimate uses for firearms.

It’s not as simple as “guns are the problem”, nor is it as simple as “more guns are the solution”, I’m afraid.