It reminds me the "Postmodernism Generator" http://www.elsewhere.org/pomo/
And the Sokal affair [1] [1] https://en.wikipedia.org/wiki/Sokal_affair
And the reverse-Sokal affair [1]
Robot-written reviews fool academics
11–20 of 26 posts
Re: Robot-written reviews fool academics
#12The quality of reviews is a general problem with the reviewing system. Just think about it: You're very busy with other work, you have many reviews to write, about papers that you're not probably really enthusiastic about, and reviews are anonymous to all but the editors. It shouldn't be surprising that this can result in rushed, bad reviews. It's also not too surprising that one can automatically generate a 'review' that looks like someone who forgot about the reviewing deadline and wrote something in a rush.
Nevertheless, while bad reviews do make it through, I do think the editors are able to recognise them for what they are.
Re: Robot-written reviews fool academics
#13I would like to see the opposite too, identify fake reviews written by robots. Otherwise people will just start using fake review robot sites so as to not spend any time.
Re: Robot-written reviews fool academics
#14"Markov chains still produce credible sentences after 30 years" reports Mark V Shaney
Re: Robot-written reviews fool academics
#15I'm not very surprised. If you've ever reviewed papers for an academic conference, you'll find that the vast majority of them are just very bad. The average ACM conference has a ~20% or so acceptance rate - and the remaining 80% isn't just a hair away from being accepted. For a big chunk of it, it's just garbage. Ill defined research problems, vague statements, poor methodology, many grammatical mistakes... given the…
Weird, I've had a very different experience reviewing for two big ACM conferences. The rejected papers are usually a bit too narrow of a contribution, but you'll usually see them appear in respected smaller venues soon after. Most of the horrible unacceptable work gets culled out in earlier passes before reviewers even see it.
Re: Robot-written reviews fool academics
#16I'm not very surprised. If you've ever reviewed papers for an academic conference, you'll find that the vast majority of them are just very bad. The average ACM conference has a ~20% or so acceptance rate - and the remaining 80% isn't just a hair away from being accepted. For a big chunk of it, it's just garbage. Ill defined research problems, vague statements, poor methodology, many grammatical mistakes... given the…
Weird, I've had a very different experience reviewing for two big ACM conferences. The rejected papers are usually a bit too narrow of a contribution, but you'll usually see them appear in respected smaller venues soon after. Most of the horrible unacceptable work gets culled out in earlier passes before reviewers even see it.
ACM has a wide range of conferences. There are conferences with an A* ranking, but also ones with a C ranking. Submissions to A*-conferences are not 80% garbage.
Re: Robot-written reviews fool academics
#17Earlier quoted context omitted.
Weird, I've had a very different experience reviewing for two big ACM conferences. The rejected papers are usually a bit too narrow of a contribution, but you'll usually see them appear in respected smaller venues soon after. Most of the horrible unacceptable work gets culled out in earlier passes before reviewers even see it.
ACM has a wide range of conferences. There are conferences with an A* ranking, but also ones with a C ranking. Submissions to A*-conferences are not 80% garbage. http://portal.core.edu.au/conf-ranks/?search=acm&by=all
Your link shows that over 75% of the conferences are B or lower. So sure, the submissions to A* conferences are not 80% garbage, but these conferences are 4% of the total. Not really representative of academic work as a whole.
Re: Robot-written reviews fool academics
#18It reminds me the "Postmodernism Generator" http://www.elsewhere.org/pomo/
and similar projects for cs and math papers
Re: Robot-written reviews fool academics
#19I'm not very surprised. If you've ever reviewed papers for an academic conference, you'll find that the vast majority of them are just very bad. The average ACM conference has a ~20% or so acceptance rate - and the remaining 80% isn't just a hair away from being accepted. For a big chunk of it, it's just garbage. Ill defined research problems, vague statements, poor methodology, many grammatical mistakes... given the…
I think the disagreements in the thread around this comment stem from the notion of an "average ACM conference" -- there is no such thing. ACM has everything from very high-quality conferences like STOC to "C-track" conferences that have a few good papers, but are mostly routine or crap.
I've done a lot of review work (but I don't like to do hard work for free any more), and nothing is more depressing than reviewing for C-track conferences. The mis-spent effort, the mis-used terminology, the buzzwords and the pretense.
Re: Robot-written reviews fool academics
#20I'm surprised this hasn't been done for math where papers actually can be completely accurate.