An Elaborate Academic-Journal Hoax • Hacker News Board

Re: An Elaborate Academic-Journal Hoax

#61

Post by amw » Thu, Oct 04, 2018, 2:32 AM UTC

davemp wrote: post #16

eindiran wrote: post #10

Earlier quoted context omitted.

That's what stuck out to me as well. This quote: "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it." Isn't the ability to separate out the quality work from the garbage the poin…

> an enterprise that relies on a widespread presumption of not-fraud This is section really gets me. How in the world is science supposed to rely on “not-fraud”? Sure if you fake your data you might dupe a diligent reviewer, but this is why results are supposed to be reproduced. Anything that relies on “not-fraud” seems inherently unscientific and straight up naive.

It's supposed to rely on not-fraud because its incentive structure is not supposed to reward gaming the system (the way that "you get money" incentivizes fraud in a market system). A certain amount of good faith is presumed because the people who are eventually found retracted en mass will have a rough time getting work in the future, and because they won't be any closer to getting work published that will make them well-known scientists in the future.

Re: An Elaborate Academic-Journal Hoax

#62

Post by amw » Thu, Oct 04, 2018, 2:34 AM UTC

mooman219 wrote: post #51

davemp wrote: post #16

Earlier quoted context omitted.

> an enterprise that relies on a widespread presumption of not-fraud This is section really gets me. How in the world is science supposed to rely on “not-fraud”? Sure if you fake your data you might dupe a diligent reviewer, but this is why results are supposed to be reproduced. Anything that relies on “not-fraud” seems inherently unscientific and straight up naive.

Exactly this. If my banking API relies on user's not trying to commit fraud instead of proper validation, then I have a very insecure API. Making a statement that your system works as long as no one tries to break it sounds almost satirical.

The result of successful banking API fraud is "I get a lot of money". The result of successful science publication fraud is, "I get some papers published, and as long as I don't get too famous and people don't look too hard at replicating my results later, maybe I get to include them in a bibliography or resume somewhere".

Re: An Elaborate Academic-Journal Hoax

#63

Post by tptacek » Thu, Oct 04, 2018, 2:41 AM UTC

Discussed earlier on HN: https://news.ycombinator.com/item?id=18127811

The hoaxers writeup is, I think, pretty misleading. Their work tended to get rejected from high-impact journals. They submitted multiple stories to some journals (notably, to Hypatia, which they brag about getting accepted in) and leave a clear impression that their most ridiculous work was what made it in, rather than a much more banal paper. They quote almost entirely favorable feedback --- including, in their earlier writeup, from papers that were ultimately rejected! They brag about papers that were ultimately rejected. They submit approximately the same paper to multiple journals and are rejected by all but the very most marginal. In at least one case, they cite the same reviewer multiple times, leaving the impression of a consensus.

If you don't think that's misleading, it's worth knowing that Yascha Mounk, cited warmly in this piece, was clearly taken in, dazzled on Twitter by a rejected paper from the hoax.

What have they really uncovered?

1. Peer review isn't replication and never could have been. Peer reviewers get, at best, a couple hours, unpaid; submitting researchers get many months, and are often grant-funded. The hoaxers took ten months to get a few papers accepted in marginal journals. The pop understanding of what peer review is is, simply, broken: peer review depends on good faith, and it was never realistic to believe otherwise.

2. There are lots of crappy journals. Didn't you already know that? The market responds to the demand that professional academics publish, by creating venues that ensure professional academics can reliably publish. That's why we have measures like "impact factors". This is not at all a phenomenon isolated to the social sciences.

If the authors even acknowledged these conclusions, they'd take all the excitement out of their hoax. We'd all nod our heads and say, "of course". Instead, they pretend to have taken down something they've dubbed "grievance studies", while pretending that they couldn't have done the same thing to, say, a marginal economics journal.

Re: An Elaborate Academic-Journal Hoax

#64

Post by tptacek » Thu, Oct 04, 2018, 2:51 AM UTC

tinalumfoil wrote: post #3

I've been more disturbed by the response to this than the papers themselves. Saying,well of course we were duped, you really expected the peer review system to not to utterly fail in the presence of a few bad actors? does not inspire my faith in the system.

What did you believe peer review was before you learned about this? You couldn't have actually done peer review before, because literally everyone who participates in that system in academia understands that they're at an immense disadvantage against a bad-faith submission. For every accepted paper, there are usually many more rejects, and all of them have to get read, and all in the reviewer's spare time.

Re: An Elaborate Academic-Journal Hoax

#65

Post by tptacek » Thu, Oct 04, 2018, 2:52 AM UTC

davemp wrote: post #16

eindiran wrote: post #10

Earlier quoted context omitted.

That's what stuck out to me as well. This quote: "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it." Isn't the ability to separate out the quality work from the garbage the poin…

> an enterprise that relies on a widespread presumption of not-fraud This is section really gets me. How in the world is science supposed to rely on “not-fraud”? Sure if you fake your data you might dupe a diligent reviewer, but this is why results are supposed to be reproduced. Anything that relies on “not-fraud” seems inherently unscientific and straight up naive.

Sussing out invalid research is the purpose of replication. A replication is itself a research project. It's not something that peer reviewers do.

Re: An Elaborate Academic-Journal Hoax

#66

Post by HarryHirsch » Thu, Oct 04, 2018, 3:07 AM UTC

ModernMech wrote: post #34

malvosenior wrote: post #18

Earlier quoted context omitted.

Alternative take: A system designed to detect bad research failed to detect blatant falsehoods, algorithmically generated noise, rabid propaganda and other seemingly easy things to detect. That PhDs worked on this doesn’t mean the system didn’t fail spectacularly.

> doesn’t mean the system didn’t fail spectacularly. You're mischaracterizing the system; it doesn't end with publication. After publication, the science goes out to a wider audience of scientists who can build on the paper or refute it if they find it's wrong. Sometimes this process can take a long time and many cycles of publicans, but that's just the nature of science. We have to stop pretending like the peer revi…

Why is this being downvoted? This is a description of the "invisible boot" as Broad & Wade call it.

Re: An Elaborate Academic-Journal Hoax

#67

Post by PinkMilkshake » Thu, Oct 04, 2018, 3:09 AM UTC

edouard-harris wrote: post #54

The authors state that this was a long term research project cut short. Which suggests the question: If you played this game long enough, undetected, could you get real scholars to cite your fake research? Future work, perhaps. (I don't have any direct evidence of this, but I strongly suspect this is an hypothesis that the authors would have wanted to test.)

I hope there is someone out there right now, working towards a degree in Critical Studies for no other reason than to get a PhD publishing nonsense and reveal it all afterwards.

Re: An Elaborate Academic-Journal Hoax

#68

Post by davemp » Thu, Oct 04, 2018, 3:10 AM UTC

tptacek wrote: post #65

davemp wrote: post #16

Earlier quoted context omitted.

> an enterprise that relies on a widespread presumption of not-fraud This is section really gets me. How in the world is science supposed to rely on “not-fraud”? Sure if you fake your data you might dupe a diligent reviewer, but this is why results are supposed to be reproduced. Anything that relies on “not-fraud” seems inherently unscientific and straight up naive.

Sussing out invalid research is the purpose of replication . A replication is itself a research project. It's not something that peer reviewers do.

Yes. My comment implies that you can get invalid research published. At least it was intended to.

The problem lies in areas where replication and formalization is difficult. There is certainly knowledge to be gained in these areas. Though calling the fields sciences, when the scientific method cannot be effectively applied, seems dubious.

The solution is not obvious, but it is clear that steps should be taken to improve rigor in the less formal (straight forward) bodies of scholarship.

Re: An Elaborate Academic-Journal Hoax

#69

Post by michaelkeenan » Thu, Oct 04, 2018, 3:26 AM UTC

blastforses wrote: post #33

lukev wrote: post #2

Representative quote: > "I am so utterly unimpressed," wrote Jacob T. Levy, a political theorist at McGill University, "by the fact that an enterprise that relies on a widespread presumption of not-fraud can be fooled some of the time by three people with Ph.D.s who spend 10 months deliberately trying to defraud it."

The whole point of a peer review system is to detect fraud. If these bullshit papers got published, who knows how many of the other papers are of equal quality.

Where did you learn that the whole point of peer review is to detect fraud?

The Wikipedia article on peer review says "Peer review...occasionally detects fraud, but is not designed to do so."[1]

The article cites a Nature article that says "Peer review is not currently designed to detect deception, nor does it guarantee the validity of research findings. It should, however, identify flaws in the design, presentation, analysis and interpretation of science and provide prompt, detailed, constructive criticism to improve research."

[1] https://en.wikipedia.org/wiki/Scholarly_peer_review

[2] https://www.nature.com/nature/peerreview/debate/nature05007....

Re: An Elaborate Academic-Journal Hoax

#70

Post by smadge » Thu, Oct 04, 2018, 3:28 AM UTC

If the author’s intention was to show the dubious quality of research in these fields, wouldn’t a better approach have been to select high impact articles from prestigious journals in the fields which they believed exemplified the problem, and then written rebuttals/critiques arguing why they are flawed?