Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI intensifies fight against ‘paper mills’ that churn out fake research

nature.com

Post reply on HN
Print view View original

111–120 of 186 posts

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#111

Post by jxramos »

There was an interesting claim I heard from Eric Weinstein regarding peer review that he characterized as a cancerous infiltration into all of science from some medical area? Aha, found it in my notes...

""" 1:32:28 Eric: but let me jump in--peer review is a cancer from outer space. It came from the biomedical community, it invaded science. The old system, because I have to say this because many people who are now professional scientists have an idea that peer review has always been in our literature and it absolutely [mff __ ] has not.

Bret: right

Eric: Okay. It used to be that the editor of a journal took responsibility for the quality of the journal which is why we had things like nature crop up in the first place because they had courageous, knowledgeable, forward-thinking editors. And so I just want to be very clear because there is a mind virus out there that says peer review is the sine qua non of scientific excellence, yada, yada, yada, bs, bs, bs. And if you don't believe me go back and learn that this is a recent invasive problem in the sciences.

Bret: recent invasive problem that has no justification for existing in light of the fact

Eric: Well not only does it have no justification for existing... When Watson and Crick did the double helix and this is the cleanest example we have. The paper was agreed should not be sent out for review because anyone who was competent would understand immediately what its implications were. There are reasons that great work cannot be peer-reviewed. Furthermore you have entire fields that are existing now with electronic archives that are not peer reviewed. Peer review is not peer review, it sounds like peer review it is peer injunction. It is the ability for your peers to keep the world from learning about your work.

Bret: keep the world from learning about your work.

Eric: because peer review is what happens, real peer review is what happens after you've passed the bs thing called "peer review". """ https://youtu.be/JLb5hZLw44s?t=5608

This is a fascinating bit of history to know about if true. I wonder if anyone has ever documented the evolution of its adoption across fields. This would be interesting too to see the HN community react to this characterization. It was the first time I ever heard of it from his wild interview with his brother.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#113

Post by juve1996 »

burkaman wrote: post #105
juve1996 wrote: post #83

Earlier quoted context omitted.

The only reason it's a problem is that papers became the measure - more papers = more prestige. The measure will simply change because now it won't have a strong signal.

That is not the only reason and it's not the most important reason. Papers are useful in the real world and people use them to learn stuff, and also reference them as evidence. If you're trying to learn about a topic and you have to wade through 90% bullshit AI papers when search for the topic, when you aren't experienced enough to immediately tell which papers are real, that's a problem. If malicious actors are able…

No, papers are not always useful. That isn't an inherent property. They can be useful. It depends on the content of the paper.

If you can't tell whether a paper is real or bullshit it doesn't matter if an AI or a human created it. Journalists will, heaven forbid, have to do real journalism instead of blindly trusting some piece of paper they found somewhere.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#114

Post by jltsiren »

eldaisfish wrote: post #107
jltsiren wrote: post #93

Earlier quoted context omitted.

The incentives of paid peer review would not be easy to get right. Especially if you want the compensation to be high enough that it would actually be an incentive to the reviewer. Many universities limit the number of hours their faculty can commit to paid external activities. Paid peer review would be one of those activities, and it would have to compete against other activities. Such as consulting, which can be ve…

This makes a good point but is largely a strawman. Review fees do not need to be this high. At $ 25 an hour, most engineering papers take about 2-5 hours to review. That’s $ 125 per reviewer or $ 500 for three reviewers, one editor and another $ 500 for publication fees. That’s $ 1,000 in total and the number of reviewers can be fewer. At the moment, greed drives open access publication fees. Journals charge thousand…

I was thinking more about something like $200 to $500/hour.

A small payment like $200 can be more of a disincentive than an incentive. You have to deal with bureaucracy to get paid and more bureaucracy to pay taxes for it. And if the money comes from a foreign source, amount of the compliance bureaucracy can be absurd. Getting paid is simply not worth it, if the payment is too small.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#115

Post by hospitalhusband »

ISL wrote: post #97

In the short term, the solution may be reputation, distributed by keychain. When author C submits a paper and they're unknown to the journal, the journal consults the keychain to see whom has vouched for C as a credible researcher. If authors A and B have vouched for them, and A and B are in similarly good standing, C is regarded as being in good stead and the editorial process can move forward as usual. If C is late…

This is web of trust.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#116

Post by dndn1 »

lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

My instinct about numbers (i.e. incl. statistics) is that they should be reproducible: on-demand, with results and workings easily subjected to interrogation.

In reality in many fields - my experience is in finance, but also in science - calculations are scattered across different languages and systems. This adds friction to any process about reproducing, understanding, analysing numbers.

This is some motivation for calculang, a language for calculations I develop. https://github.com/calculang/calculang

For the HN crew it's an under-development(!) functional language with properties to permit flexible designs that can scale. It's for numbers and if you share the model alongside your numbers people can check them according to the model and see the workings.

It goes well with visual number [dev/person]tools that I will release one of soon, and in the future watch for a browser extension for the workings behind numbers you are reading.

Important, it won't address the raw data part of the problem, but where numbers following from that are concerned, it might get closer to that instinct.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#117

Post by riku_iki »

sebzim4500 wrote: post #29
riku_iki wrote: post #17

Earlier quoted context omitted.

There is a good metric in AI research: reproducible results on well known benchmarks (Big Bench for example).

There's no good benchmark that tests for most of the issues with LLMs, for example. Maybe if we had a benchmark that told you how often a model hallucinated we would have solved the problem by now.

There are plenty of benchmarks with factual questions.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#118

Post by notahacker »

huijzer wrote: post #61
CobrastanJorji wrote: post #51

Earlier quoted context omitted.

No, science definitely goes worse when there are more bad papers. There's a popular argument about freedom of speech that comes down to "the solution to bad speech is more good speech, not limiting bad speech," but that only works (assuming it's true at all) when there's some way for a listener to discern truth with some research or effort. > "Good papers are good because they have good arguments" No they are not. Go…

> There's a popular argument about freedom of speech that comes down to "the solution to bad speech is more good speech, not limiting bad speech," but that only works (assuming it's true at all) when there's some way for a listener to discern truth with some research or effort. Yes exactly. If it is too hard for a group of scientists to figure out a flaw, then it is also too hard for two peer reviewers to figure out…

> If it is too hard for a group of scientists to figure out a flaw, then it is also too hard for two peer reviewers to figure out a flaw.

The issue isn't that flawed papers are necessarily hard to spot, the issue is that indiscriminate publishing means that anyone studying a topic has to wade through a lot of flawed ones before they find anything halfway useful. Time and attention are finite. Peer review and journals caring about reputation, in theory, caps the number of readers of rubbish papers at 2 (and disincentivises writing unpublishable crap to submit in the first place... although this is now eroded since generating rubbish papers is now effort-free)

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#119

Post by burkaman »

juve1996 wrote: post #113
burkaman wrote: post #105

Earlier quoted context omitted.

That is not the only reason and it's not the most important reason. Papers are useful in the real world and people use them to learn stuff, and also reference them as evidence. If you're trying to learn about a topic and you have to wade through 90% bullshit AI papers when search for the topic, when you aren't experienced enough to immediately tell which papers are real, that's a problem. If malicious actors are able…

No, papers are not always useful. That isn't an inherent property. They can be useful. It depends on the content of the paper. If you can't tell whether a paper is real or bullshit it doesn't matter if an AI or a human created it. Journalists will, heaven forbid, have to do real journalism instead of blindly trusting some piece of paper they found somewhere.

> If you can't tell whether a paper is real or bullshit it doesn't matter if an AI or a human created it.

That's true, but now it's incredibly easy to create a human-quality bullshit paper. That wasn't true a year ago. It doesn't matter who created it, but it does matter how easy it was to create.

> Journalists will, heaven forbid, have to do real journalism instead of blindly trusting some piece of paper they found somewhere.

That's true, but now it's incredibly easy to generate a real-looking piece of paper and send it to a journalist, prompting them to spend a day doing real journalism to verify it. That wasn't true a year ago.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#120

Post by notahacker »

AnthonyMouse wrote: post #101
webnrrd2k wrote: post #32

Earlier quoted context omitted.

Re: "Science does not get worse when there are more bad papers"... I don't think that this is true at all. Weeding through bad papers is, at a minimum, an opportunity cost, as is a good paper built on top of a bad one. Also, there is a societal cost in that bad research can get picked up and believed by people, like the anti-vax crowd. Or, bad research can be used to push an agenda, like anti-climate change.

The solution to this is reputation. The flaw is that we're using the reputation of for-profit journals rather than the research institutions. You shouldn't have confidence in a paper because it was published in Nature, you should have confidence in it because it was published by Harvard or the University of California or Google Research, who puts the name of their institution on it and thereby stakes their reputation…

The research institutes are also for profit bodies, just much more of an closed shop because there's only so many people they can employ at any one time (and that's before we even get into tenure or the fact all you're really doing is shifting the peer review burden onto people that report into the person that wrote the paper).

You can have papers published by Google Research or by nobody now: turns out academics like the journals' attempts to curate interesting stuff more than only ever reading stuff on their bookmark list or that's mailed to them in a desperate attempt to get some engagement. And for all that they might not like the long delays and low chance of success associated with submitting to Nature, they like the idea of trying to persuade an academic at an elite institution to go to the trouble of replicating their research as the only means to build their reputation even less...

Post reply on HN