An Elaborate Academic-Journal Hoax

Mounk didn't read the hoaxers writeup carefully enough to be a reliable judge; he gleefully tweeted two papers from the hoax that were rejected everywhere they were submitted, as evidence of the important problem the hoax uncovered --- this despite the fact that the hoaxers did in fact provide an accounting of what got accepted where (even though their writeup at the beginning of the piece misrepresented that accounting).

From what Pinker wrote, who knows if he even read it?

If the particular research is non-falsifiable and the critic is convinced the entire field is not an academic discipline at all but a cult with its entirely own internal logic with no relation to reality, how would an objective rebuttal be constructed? Is the right way for atheists to engage with religious scholars to critique the most nonsensical output of theologians?

It's hard to tell specifically how their methodology went, but this bit of the author's write-up jumps out:

> As for our performance, 80% of our papers overall went to full peer review, which keeps with the standard 10-20% of papers that are “desk rejected” without review at major journals across the field. We improved this ratio from 0% at first to 94.4% after a few months of experimenting with much more hoaxish papers.

This makes it sound like the authors wrote a bunch of absurd articles, all of which got desk rejected, and then they iterated until they got papers accepted somewhere. With that methodology, the conclusion ("we got papers accepted somewhere!") doesn't seem very surprising.

Edited to add: if the above is correct and their early submissions were all rejected, one could reasonably say of the results: "Editorial standards at all journals surveyed were more rigorous than the authors initially expected".

Peer Review is not set up to detect fraudulent research and attempts to do so are fraught with risk, since established researchers often have a hard time believing in results from new paradigms. What the heck is an author supposed to do if a reviewer says "I think these numbers are made up"?

The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough that the community can precisely understand what happened, and whether it has methodological issues that the authors may have missed. But we always approach reviews with a charitable lens, even when reviewing for extremely low-acceptance outlets.

The process that detects the bad research comes afterwards, as it should.

> To answer these questions, this article engages feminist geography and broader feminist literature and draws on nearly 1000 h of public observations of dogs and their human companions conducted at three dog parks in Southeast Portland, Oregon, beginning on 10 June 2016, and ending on 10 June 2017.

In Sokal's single hoax in 1996 he (and his buddies, I guess) made a claim, followed it by a non-sequitur of true but uninteresting sentences about physics, then repeated the claim. It was a sandwich with no meat in it, and the (small, non-peer-reviewed) post-modern journal ate it up.

Sokal wanted to show that a certain style of post-modern writing substituted scientific jargon for an argument (or at least it used references to esoteric scientific fields as pretentious, uninteresting metaphors). The elegance of parodying this style was that his submission contained no data at all. It wasn't merely that the conclusion of the submission wouldn't be replicated. It was that any conscious human should notice the complete lack of supporting evidence for the claim!

The upshot of the journal accepting the article was that "the emperor had no clothes," so to speak. Sokal went on to write a book showing how several prominent post-modern thinkers used scientific non-sequiturs to make their non-scientific ideas sound more weighty. (Just as he had in his hoax.)

These three authors, however, are going a different route from Sokal-- specifically, one that provides fake data as evidence of a claim. (Or I guess they could have actually sat at dog parks for 1000 hours, but it doesn't matter for the point I'm making.) By providing fake data, this hoax passes an initial filter which the Sokal hoax did not-- namely, the filter that should reject submissions which lack any supporting evidence whatsoever. That means this hoax only reveals a lack of replication and/or verification, whereas the Sokal hoax reveals a lack of both peer review and basic reading comprehension even by a non-physicist. That makes this a weaker class of hoax than the Sokal hoax.

Finally, the authors purport to be going after an entire "ideological slant" in academia. Whatever one thinks of that aim, it is enormous and amorphous when compared to the narrow aim of Sokal keeping scientific jargon from being misused/misplaced.

So to recap:

* 1996: strong hoax successful against a single pomo journal, weak but persuasive claim against coopting scientific jargon for non-scientific aims

* 2018: weak hoax successful against 7 of (total number?) journals, strong and unpersuasive claim against all leftist academic fields

Now I'm curious-- did anyone in the late 90s criticize Sokal's hoax on the grounds that his careful constraints would be ignored by future pranksters?

Edit: clarification