Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Elaborate Academic-Journal Hoax

chronicle.com

Post reply on HN
Print view View original

91–100 of 165 posts

Re: An Elaborate Academic-Journal Hoax

#91

Post by tomlock »

darawk wrote: post #85
tomlock wrote: post #83

Earlier quoted context omitted.

> Is that so? So, you'd have no problem getting a nonsense paper accepted into a math, computer science, physics or even economics journal? I would be extremely surprised if you could pass of work of such low quality in journals of comparable impact factor in any of these disciplines. https://en.wikipedia.org/wiki/Replication_crisis > A 2016 study in the journal Science found that two-thirds of 18 experimental studie…

The replication crisis is an entirely different thing. I'm not sure what point you're trying to make.

It's entirely possible to get a nonsense paper into an economics journal, even when not intentionally trying to expose the discipline as "nonsense". I'm not sure what you're not understanding here.

Re: An Elaborate Academic-Journal Hoax

#92

Post by tomlock »

darawk wrote: post #86
gbhn wrote: post #82

Earlier quoted context omitted.

I read over a couple of the papers. The descriptions of them by the authors did not match the content of the papers. For instance, the one they call "Hooters" is described as "A gender scholar goes to Hooters to try to figure out why it exists." The actual paper is written as a record of transcripts of visits to a restaurant and an extraction of particular conversation themes. At no point did I see the paper question…

> I read over a couple of the papers. The descriptions of them by the authors did not match the content of the papers. So did I. The dog park paper more or less matches its description. The fat bodybuilding paper is exactly as described. The dildos paper is as described. Do you have any descriptions aside from the Hooters one that you take issue with? > Yes, the methodology isn't that great, and the paper was rejecte…

Could you reference some texts from within gender studies that you feel are both representative of the field, and which outline the negative aspects of the epistemology of gender studies that you are perceiving?

Re: An Elaborate Academic-Journal Hoax

#93

Post by fenomas »

darawk wrote: post #81
tptacek wrote: post #63

Discussed earlier on HN: https://news.ycombinator.com/item?id=18127811 The hoaxers writeup is, I think, pretty misleading. Their work tended to get rejected from high-impact journals. They submitted multiple stories to some journals (notably, to Hypatia , which they brag about getting accepted in) and leave a clear impression that their most ridiculous work was what made it in, rather than a much more banal paper. Th…

> The hoaxers writeup is, I think, pretty misleading. Their work tended to get rejected from high-impact journals. What's your evidence for that? A number of their papers got accepted into decently high impact journals, including Hypatia. > They submitted multiple stories to some journals (notably, to Hypatia, which they brag about getting accepted in) and leave a clear impression that their most ridiculous work was…

> You seem to think that the papers that got accepted were 'banal', but that doesn't change the fact that they were accepted.

The banality of the papers is the central weakness of the whole thing. The whole point of Sokal- or SCIgen-style hoaxes is that the papers were gibberish - so when a journal publishes one, the author can legitimately say "Hey look, this journal published gibberish!".

The authors here can't say anything like that. They wrote real papers, arguing real conclusions they thought the journals might agree with. Then those papers got rejected, so they toned it down and wrote more, and then they got peer-review feedback so they changed things in response, etc. The most one can fairly say about the results is "Hey look, these journals published papers that we think are really bad". Which is fair enough, but surely they could say the same thing about most of what those journals already publish, right?

The liberty of the whole thing is writing it all up as if a Sokal-style "we got them to print gibberish" narrative had occurred, when that seems clearly not to be the case.

Re: An Elaborate Academic-Journal Hoax

#94

Post by foldr »

davemp wrote: post #68
tptacek wrote: post #65

Earlier quoted context omitted.

Sussing out invalid research is the purpose of replication . A replication is itself a research project. It's not something that peer reviewers do.

Yes. My comment implies that you can get invalid research published. At least it was intended to. The problem lies in areas where replication and formalization is difficult. There is certainly knowledge to be gained in these areas. Though calling the fields sciences, when the scientific method cannot be effectively applied, seems dubious. The solution is not obvious, but it is clear that steps should be taken to impr…

It isn't possible to attempt replication of every scientific paper that's published. A lot of people on this thread seem to have difficulty accepting this fact, but the time and money to do that simply isn't available.

If a large percentage of scientists were to engage in deliberate fraud, there wouldn't be any feasible way to regulate that problem away. All available resources would end up being spent on scientists investigating each other for fraud, and no actual science would get done. Bear in mind that in many cases, establishing that an experiment is not fraudulent could require more time and effort than the original experiment!

As with many other human institutions, we have no choice but to rely on most people being honest most of the time.

Re: An Elaborate Academic-Journal Hoax

#95

Post by int_19h »

I see a lot of people complaining that it is done in bad faith, and doesn't prove much.

But is there really much to prove? What's the baseline? I mean, we're talking about a field where this got published - and it was not a hoax:

http://journals.sagepub.com/doi/full/10.1177/030913251562336...

The people behind this hoax had an unenviable task of matching and beating a paper like that (since, apparently, by itself it was not sufficient to prove the point). How would you do so?

Re: An Elaborate Academic-Journal Hoax

#96

Post by foldr »

int_19h wrote: post #95

I see a lot of people complaining that it is done in bad faith, and doesn't prove much. But is there really much to prove? What's the baseline? I mean, we're talking about a field where this got published - and it was not a hoax: http://journals.sagepub.com/doi/full/10.1177/030913251562336... The people behind this hoax had an unenviable task of matching and beating a paper like that (since, apparently, by itself it…

Bad papers get published in all fields. Linking to one bad journal article shows nothing.

Here, for example, is a reference to a bad paper published in a field which is not usually the target of this kind of criticism:

https://academia.stackexchange.com/questions/9602/rediscover...

https://math.berkeley.edu/~ehallman/math1B/TaisMethod.pdf

Re: An Elaborate Academic-Journal Hoax

#97

Post by ModernMech »

pochamago wrote: post #49
ModernMech wrote: post #34

Earlier quoted context omitted.

> doesn’t mean the system didn’t fail spectacularly. You're mischaracterizing the system; it doesn't end with publication. After publication, the science goes out to a wider audience of scientists who can build on the paper or refute it if they find it's wrong. Sometimes this process can take a long time and many cycles of publicans, but that's just the nature of science. We have to stop pretending like the peer revi…

What is the point of journals if quite literally anything can be published in them? Isn't the point that they serve as a basic filter for obvious noise? Shouldn't we be critical of a peer review process that doesn't function at even the most basic level?

> quite literally anything can be published

I've got a stack of rejected papers that beg to differ.

> Isn't the point that they serve as a basic filter for obvious noise?

Yeah, and obvious noise is usually rejected. But sometimes it is not so obvious what the noise is, and that's why the scientific process is bigger than the peer review system.

> Isn't the point that they serve as a basic filter for obvious noise?

Totally, there is a lot to fix about it. But if you really want to put your effort into changing the system, I don't think you should focus on peer review. Destroying the business model of journals is where I would start.

Re: An Elaborate Academic-Journal Hoax

#98

Post by malvosenior »

UncleMeat wrote: post #78
malvosenior wrote: post #18

Earlier quoted context omitted.

Alternative take: A system designed to detect bad research failed to detect blatant falsehoods, algorithmically generated noise, rabid propaganda and other seemingly easy things to detect. That PhDs worked on this doesn’t mean the system didn’t fail spectacularly.

Peer Review is not set up to detect fraudulent research and attempts to do so are fraught with risk, since established researchers often have a hard time believing in results from new paradigms. What the heck is an author supposed to do if a reviewer says "I think these numbers are made up"? The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough tha…

> The purpose of peer review is to decide whether a paper is useful to the community, whether the presentation is good enough that the community can precisely understand what happened, and whether it has methodological issues that the authors may have missed.

It clearly failed to do that in this case.

Re: An Elaborate Academic-Journal Hoax

#99

Post by malvosenior »

UncleMeat wrote: post #79
pervycreeper wrote: post #73

Earlier quoted context omitted.

>The point of peer review is to stop bad research, not fake research I would submit that "fake research" is, in fact, bad research.

To a reviewer with three hours to spare, fake research is indistinguishable from good research. Unless you want to give me months to review a paper, I will not be able to detect fraud. Instead I rely on the fact that the community will attempt to integrate these ideas after publication and arrive at the conclusion that the paper was garbage.

If you only have three hours, can't detect fraud, can't tell if the the paper is even real. Then what's the point of even doing the peer review? Again, peer review offered no value at all in this case.

Re: An Elaborate Academic-Journal Hoax

#100

Post by tptacek »

darawk wrote: post #81
tptacek wrote: post #63

Discussed earlier on HN: https://news.ycombinator.com/item?id=18127811 The hoaxers writeup is, I think, pretty misleading. Their work tended to get rejected from high-impact journals. They submitted multiple stories to some journals (notably, to Hypatia , which they brag about getting accepted in) and leave a clear impression that their most ridiculous work was what made it in, rather than a much more banal paper. Th…

> The hoaxers writeup is, I think, pretty misleading. Their work tended to get rejected from high-impact journals. What's your evidence for that? A number of their papers got accepted into decently high impact journals, including Hypatia. > They submitted multiple stories to some journals (notably, to Hypatia, which they brag about getting accepted in) and leave a clear impression that their most ridiculous work was…

I think you've misread me.

My own comment leads off by acknowledging they got a paper accepted into Hypatia --- my gripe is that it's not the lurid farce they suggest they got accepted (a paper demanding that white male students sit on the floor in chains in class) but rather a much more boring paper about the legitimacy of hoaxes like theirs.

When I said "They submit approximately the same paper to multiple journals and are rejected by all but the very most marginal", I was referring to a specific paper (in fact, the Mein Kampf paper), not to all the papers. My point was: they tried that paper in several places, and the (academically) reputable venues rejected them.

Yes, I think --- I'm actually pretty confident --- that I could get a false paper accepted in a CS venue. Economists have already remarked on Twitter that you don't have to get "nonsense" papers accepted yourself, because the literature is already full of what turns out to be nonsense (do keep in mind that in large part what these hoaxers submitted was not nonsense, but rather fabricated research including field work).

Post reply on HN