Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI intensifies fight against ‘paper mills’ that churn out fake research

nature.com

Post reply on HN
Print view View original

131–140 of 186 posts

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#131

Post by godelski »

omginternets wrote: post #110
godelski wrote: post #63

Earlier quoted context omitted.

On the other hand, maybe we'll abandon peer review and return to a pre 1950's like style? Everyone in ML just uses arxiv because the field is moving so fast. All work is realistically peer reviewed once it is out there. We treat conferences (more important than journals) as very noisy signals. Unfortunately, we still use those as metrics for completing degrees or hiring people. The way I would try to make this better…

I think peer review exists for reasons beyond building or affirming the reputation of a scholar. In particular, it’s regarded as a filter for eliminating the very worst-quality research. And judging from the manuscripts that came across my desk when I was still in academia, I’d say it’s performing this particular duty reasonably well. As such, the connection between AI fakery and peer-review isn’t all that clear cut…

I mostly agree with your assessment of peer review, but feel that it's gone too far. It's decades of metric hacking going unresolved. My research area is ML and it is definitely far out of hand. When I was in physics I'd probably be closer to your statement. But right now we have measurements that conclude "reviewers are good at identifying bad papers, but bad at identifying good papers." Either I think we need to accept more works (as long as they are valid and useful) and/or reducing the value of a top tier venue (which right now is top or none). I think papers should get multiple rounds, as our goal is to improve works, and submissions should always be rolling. I think this can help, because I agree that peer review should be about eliminating the worst, not identifying the best (far too difficult, and hundreds of years of evidence).

I think one important thing that venues could do is host data and code/tools for works. Harddrive space is rather cheap now (especially with tax benefits and donations) and can only result in high value to the community. The other thing is having formats like OpenReview, where works can continuously be discussed in the open. With authors and others being able to defend/criticize/question works. But I think there should be some filter, even if low, and rules for control of quality.

For the most part, I do actually think getting rid of venue based review would be a step in the right direction. I use these words because I like to encourage the idea that open publication still leads to peer reviewing. In ML a lot is done with arxiv + twitter, I just think we should better formalize this. It allows for a lot of freedom and for high speed. It has problems, but I don't see them as any worse than the current system (I see an overall decrease in problems). Good science requires risks and a lot of creativity. The modern word advantage is that we have higher numbers of researchers and monte carlo sampling in parallel helps optimize. You want to encourage tail end samples too to escape local minima. The history of science is a history of upsets. You don't get upsets by doing what everyone is doing. I agree that top down hierarchies discourage such thinking and are an overall net negative to science and advancing human knowledge. Creativity is critical.

For predictions, I agree. I lean towards predicting disruption, but I'm not even certain of that. And disruption can go many different ways. I'm glad we're starting the discussions, but I think they need to be deeper and more honest.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#132

Post by mike_hearn »

RosanaAnaDana wrote: post #127
lacker wrote: post #35

This seems like an extension of the replication crisis. In many fields, most published research is already bogus. The idea that peer review is enough to ensure that research is valid is perhaps not scaling well. It would be great to have things like open data sharing. At least in astronomy, which I'm somewhat familiar with, it doesn't seem like we're that close. Most scientists cannot even reproduce their own results…

I think if the data is valid, open access, and reproducible, it shouldn't matter if the paper was written by AI.

Unfortunately this assumes that the only thing which can go wrong is lack of reproducibility. Not so. I read a lot of public health papers during COVID and a staggering quantity (IMHO nearly all of them) should not have been published; many of them would have been reproducible despite that.

Other things that can and do regularly appear in reproducible, peer reviewed papers:

• Nonsensical methodologies

• Logical fallacies

• Mis-representation of their data

• Incorrectly implemented software

• Source datasets that are cherry-picked

One might think that things like incorrectly implemented software would fall under the umbrella of irreproducible research, but that won't work. Some fields don't really recognize a distinction between model implementation and specification. The model is the implementation and if a description was once published it's quite possibly either too vague to implement, out of date, wrong, or all three. IIRC Prof Neil Ferguson's team actually rejected an attempted replication of one of their epidemic models on the basis that only they were qualified to use it! Pseudo-science like this goes unremarked in universities, only outsiders seem to care.

Sometimes you get an honest academic who knows what they're supposed to do and actually does it, but there's no observable benefit to them from doing so because the ones who don't bother don't seem to suffer any consequences.

tl;dr The biggest problem with the replication crisis is the framing of it as being about replication. What the world actually faces is a misleading research crisis. You can drive the replicability rate to 100% and you'll still find whole fields consisting of logical fallacies and misinformation.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#133

Post by mike_hearn »

tga_d wrote: post #121
JacobThreeThree wrote: post #71

Earlier quoted context omitted.

The current scientific system has long been known to have serious problems of incorrect results and conflicts of interest. This article seems like an attempt to pin the crisis on an AI scapegoat. From 2015, the Editor of The Lancet: The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exp…

How do you see this as scapegoating? The headline specifically says "intensifies", the article very clearly positions AI-fabricated data as an extension of existing problems, and I don't see anything in the article downplaying those existing problems (the entire closing section is about how the summit was on issues broader than AI).

Nature's reporting on the problem of paper mills is surprisingly high quality and honest, given that these reports directly attack the credibility of Nature itself (and many other journals).

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#134

Post by godelski »

stubybubs wrote: post #82
godelski wrote: post #63

Earlier quoted context omitted.

On the other hand, maybe we'll abandon peer review and return to a pre 1950's like style? Everyone in ML just uses arxiv because the field is moving so fast. All work is realistically peer reviewed once it is out there. We treat conferences (more important than journals) as very noisy signals. Unfortunately, we still use those as metrics for completing degrees or hiring people. The way I would try to make this better…

It's easy to do this with anything in computer science as others can implement or recreate whatever is being discussed easily. Not so with a 6 month experiment with several groups of humans, or even a 2 week one with rats.

This is not always true. ML has reproducibility issues despite the status quo being open models. Even if we don't include proprietary datasets there are still issues. Compute is one issue, but we can even ignore that[0]. The Lottery Ticket[1,2] plays a big role, in that you can just get lucky. This really should have resulted in treating benchmarks as weaker indicators but see other comments about reviewing incentives. Another issue is that it is status quo to optimize hyperparameters on test data and this results in information leakage. While this won't affect results of running a checkpoint there are issues in reproducing the work. It also adds noise. Generative papers also have a large issue in not showing random uncurated samples, which introduces huge biases and we can argue this is a reproducibility issue as you can't reproduce the results show in the works. Anyone that's played with things like Stable Diffusion (or any generative model) will be familiar with this.

There are more nuanced issues and things that require intimate domain knowledge to fully understand, but I wanted to push back at this comment because I see a lot of people just brush off reproducibility concerns by pointing to GitHub. While it helps, it definitely doesn't make reproducibility "easy" or concerns nonexistent.

[0] Sometimes we can't though as scaling has more effects than obvious. e.g. GANs can't scale batch on a per GPU level but scaling by multiple GPUs/nodes does give an advantage in quality (not just training times). This is non-obvious and many things can play a role.

[1] https://arxiv.org/abs/1803.03635

[2] https://arxiv.org/abs/2109.08203

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#135

Post by mike_hearn »

SantalBlush wrote: post #69
huijzer wrote: post #58

Earlier quoted context omitted.

> noise and conflicted interests dominate the conversation and drown out more rigorous or valuable information. That's actually a great point. Who says this isn't already the case? There isn't much evidence to suggest that things have gotten better after the Why Most Published Research Findings Are False paper in 2005. I think people should be extremely skeptical of anything they read, irregardless of a peer review s…

>I think people should be extremely skeptical of anything they read This is a mistake that many HN readers make; they think that if one is equipped with some above-average level of intelligence, one can discern the validity of new research. But this is wrong. It usually takes many years of study before one can begin to clearly understand what is even being said, let alone whether it has any veracity. People of above-…

In practice that's not the case. Maybe for some fields it is, but there are definitely plenty of fields where minimal expertise is required to do basic sanity checks of a paper. And there are many generic 'tells' that are indicative of issues across a wide range of fields, e.g. if a paper is reporting mostly P = 0.049 results, well ...

As an example of an issue that doesn't require much expertise to spot, I was reading a health paper a couple of days ago that reported on an intervention in a specific population. The paper said words to the effect of "[the intervention] was effective, especially for men". The associated chart not only had (somehow?!) got a legend that didn't match the actual chart lines, but whilst the line for men did indeed go down the line for women was very clearly flat! This should have been reported as "no effect in women but an effect in men" but wasn't. When the wording doesn't match the data being reported, that's a good sign in any field that the researchers know they're treading on thin ice. That particular claim was a correlation/causation fallacy anyway, which is very common in health. They didn't have any proof it was their intervention causing the reduction and there were a bunch of reasons to suspect it wasn't. But the intervention was long term and high effort so it's not a surprise they wanted to find something.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#136

Post by YeGoblynQueenne »

huijzer wrote: post #14

Instead of fighting against "paper mills", let’s fight against journals. There are strong arguments against the need for peer-review for example [1]. Science does not get worse when there are more bad papers, science gets better when there are more good papers. Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening. I’m not saying that is because the lack of revie…

>> Most papers in AI aren’t even reviewed by peers or editor and guess what: we have lots of progress happening.

If by progress you mean a constant creep of SOTA on meaningless benchmarks, then yeah, we have "progress", but that progress is a strange kind of progress that is measured only on its own, self-selected criteria, and that does nothing to advance the total sum of knowledge. The main beneficiaries of this "progress" are large technology corporations who have now taken over research in AI and are turning it to profit. There has definitely been proress in money-making schemes and personal aggrandisement schemes of charlatans and mountebanks, in AI, aplenty.

If by progress you meant scientific progress, then, no, there has been none of that in recent years. It is questionable if there has ever been any kind of scientific progress associated with AI. While early pioneers of AI, like John McCarthy and Claude Shannon, wished to establish a scientific programme of research in human and machine intelligence, with the purpose of understanding the former by developing the latter, this programme was soon set aside, and activity concentrated instead in what McCarthy used to call the "look ma, no hands disease of AI":

>> Much work in AI has the ``look ma, no hands'' disease. Someone programs a computer to do something no computer has done before and writes a paper pointing out that the computer did it. The paper is not directed to the identification and study of intellectual mechanisms and often contains no coherent account of how the program works at all.

http://www-formal.stanford.edu/jmc/reviews/lighthill/lighthi...

And this is where we still are today. AI is no science.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#137

Post by mohave529 »

From my perspective as an outsider, I have always been amazed by the use of papers in academic research as a means of communicating findings to the wider world. I find it problematic that these papers are often formatted in a way that makes them highly unreadable, with two columns and compressed text. In my opinion, adopting more modern methods of publishing research could greatly enhance the overall quality of research by making papers more accessible and increasing the likelihood of them being read.

Imagine a scenario where there is a standardized format for academic papers, where the conclusion is explicitly derived from specific data and accompanied by confidence intervals. This standardized schema would enable easier referencing of other papers and easy incorporation of additional data through features like autocomplete. Implementing such a system could potentially reverse the trend of academic papers that use excessive and unnecessary language to appear more intellectually rigorous, even when the actual information being conveyed is limited.

By embracing these changes, we could create a more transparent and efficient research environment that promotes clearer communication and enhances the impact of academic findings.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#138

Post by SantalBlush »

mike_hearn wrote: post #135
SantalBlush wrote: post #69

Earlier quoted context omitted.

>I think people should be extremely skeptical of anything they read This is a mistake that many HN readers make; they think that if one is equipped with some above-average level of intelligence, one can discern the validity of new research. But this is wrong. It usually takes many years of study before one can begin to clearly understand what is even being said, let alone whether it has any veracity. People of above-…

In practice that's not the case. Maybe for some fields it is, but there are definitely plenty of fields where minimal expertise is required to do basic sanity checks of a paper. And there are many generic 'tells' that are indicative of issues across a wide range of fields, e.g. if a paper is reporting mostly P = 0.049 results, well ... As an example of an issue that doesn't require much expertise to spot, I was readi…

This is a great example of what I'm talking about. P-values of 0.049 is not a "tell"; that is an HN meme. You can have p-values above 5 percent and the paper can still convey novel and valid research, depending on the nature of the study. The 5-percent value is a common cutoff, but it is a largely arbitrary one. Sometimes there is justification for a higher one, and sometimes it should be lower.

To gauge what sort of p-value suggests a useful model requires familiarity with the area and the broader context of the research.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#139

Post by jltsiren »

godelski wrote: post #63
omginternets wrote: post #57

Prediction: as a result of fake research and it’s ilk (fake credentials, fake trade-journal pubs, fake reviews, etc.), we will observe a renewed interest in referrals and meatspace networking. It will become correspondingly difficult for outsiders to enter professional circles.

On the other hand, maybe we'll abandon peer review and return to a pre 1950's like style? Everyone in ML just uses arxiv because the field is moving so fast. All work is realistically peer reviewed once it is out there. We treat conferences (more important than journals) as very noisy signals. Unfortunately, we still use those as metrics for completing degrees or hiring people. The way I would try to make this better…

I don't think ML is a good example, because it's an outlier within an outlier.

CS is already a weird field, because it leans so heavily on conference papers that see a single round of reviews. Journal papers with multiple rounds of reviews are more common in other STEM fields. Because of this difference, peer review in CS is more focused on accepting/rejecting the paper than on improving it.

And then the CS model breaks down in ML, which is far too popular for its own good. There are too many people working on the same topics, writing too many papers and submitting them to too few conferences, which have too many people attending. CS conferences are supposed to be more like community meetings than formal conferences. Informal discussions are the main point, but that doesn't scale beyond a few hundred attendees.

Re: AI intensifies fight against ‘paper mills’ that churn out fake research

#140

Post by mike_hearn »

SantalBlush wrote: post #138
mike_hearn wrote: post #135

Earlier quoted context omitted.

In practice that's not the case. Maybe for some fields it is, but there are definitely plenty of fields where minimal expertise is required to do basic sanity checks of a paper. And there are many generic 'tells' that are indicative of issues across a wide range of fields, e.g. if a paper is reporting mostly P = 0.049 results, well ... As an example of an issue that doesn't require much expertise to spot, I was readi…

This is a great example of what I'm talking about. P-values of 0.049 is not a "tell"; that is an HN meme. You can have p-values above 5 percent and the paper can still convey novel and valid research, depending on the nature of the study. The 5-percent value is a common cutoff, but it is a largely arbitrary one. Sometimes there is justification for a higher one, and sometimes it should be lower. To gauge what sort of…

That's the point - it's an arbitrary cutoff, so the high frequency with which results cluster around that point is indicative of a problem.

If a field accepts P=0.05 as significant it means 1 in 20 results can be false positives just by random chance. Now think about how many papers get published, and many of them report more than one thing. The right threshold should really be an order of magnitude lower. It's not OK for scientists to report FPs at that rate, and that's a big part of the reason for declining confidence in science.

Post reply on HN