Something similar happened with IEEE[1] - It had accepted approx 120 papers generated by SCIgen[2].
[1] - http://www.nature.com/news/publishers-withdraw-more-than-120...
[2] - http://pdos.csail.mit.edu/scigen/
Half the papers at NIPS would be rejected if the review process were rerun
21–30 of 55 posts
Re: Half the papers at NIPS would be rejected if the review process were rerun
#22Earlier quoted context omitted.
Can you say more about why you believe this is true?
What does "poor quality" mean? There is no absolute standard for quality. "Poor" is something like "less good than usual compared to the recent work in this community". So the top-scoring third-ish of papers sent to the currently-converged-on favourite venue of a community are pretty much by definition not poor. Unless something very weird indeed happens one year. There are usually only very few really excellent pape…
Ok, we've gone from "top tier venue, basically impossible to have a large fraction of poor papers submitted" to "Most papers are filler in retrospect" and "conferences need to accept a decent number of papers so that people will show up and cover the costs of the meeting". I guess if I am deciding whether or not to hire a professor I would be tempted to disregard publications in this conference.
Re: Half the papers at NIPS would be rejected if the review process were rerun
#23For those who didn't read the story: NIPS decided to run this experiment themselves to identify problems.
That's a a really brave thing to do, and deserves serious credit.
Re: Half the papers at NIPS would be rejected if the review process were rerun
#24I am surprised the committees were "tasked with a 22.5% acceptance rate". Couldn't more than 77.5% of the submissions have been of poor quality?
When I've arranged conferences, we had a certain number of time slots. It's a bit flexible, in that we can decide to allocate a longer time for two talks, or shorter time for three, depending on the talks.
It could be that they had a first pass at a schedule, used that to set a first cut for the reviewers, then adjusted the schedule once they figured they needed to add another 42 papers.
Also, not being accepted does not mean that a paper is poor. They used a rank system, so it only means that others had papers which appeared to be better.
Re: Half the papers at NIPS would be rejected if the review process were rerun
#25Something similar happened with IEEE[1] - It had accepted approx 120 papers generated by SCIgen[2]. [1] - http://www.nature.com/news/publishers-withdraw-more-than-120... [2] - http://pdos.csail.mit.edu/scigen/
IEEE may be more prone to precision errors (letting bad papers in) while NIPS may be prone to recall errors (throwing good papers out).
With the way reviewing is done (no one can take a week off to read and fully comprehend the four papers they are given) you cannot achieve perfect separation - even if that were possible.
Re: Half the papers at NIPS would be rejected if the review process were rerun
#26Earlier quoted context omitted.
And they look fucking awful.
They're supposed to look hastily-made to reflect the imprecision in the estimates. https://www.chrisstucchio.com/blog/2014/why_xkcd_style_graph...
I love the general idea of comic-style graphs, but this particular implementation of it does indeed look "fucking awful" in my opinion. The graph itself is fine, but the axis is made of these faux wavy line that eerily repeats itself.
Re: Half the papers at NIPS would be rejected if the review process were rerun
#27Earlier quoted context omitted.
What does "poor quality" mean? There is no absolute standard for quality. "Poor" is something like "less good than usual compared to the recent work in this community". So the top-scoring third-ish of papers sent to the currently-converged-on favourite venue of a community are pretty much by definition not poor. Unless something very weird indeed happens one year. There are usually only very few really excellent pape…
Ok, we've gone from "top tier venue, basically impossible to have a large fraction of poor papers submitted" to "Most papers are filler in retrospect" and "conferences need to accept a decent number of papers so that people will show up and cover the costs of the meeting". I guess if I am deciding whether or not to hire a professor I would be tempted to disregard publications in this conference.
It depends what you are optimizing for.
Number of publications is a proxy for how much funding a professor can generate. Not much else.
> "Most papers are filler in retrospect" and "conferences need to accept a decent number of papers so that people will show up and cover the costs of the meeting"
None of these are conflicting. Conferences are often more about networking than the papers. Many paper are filler, but often only in retrospect. They are not obviously filler when presented.
Re: Half the papers at NIPS would be rejected if the review process were rerun
#28Earlier quoted context omitted.
And they look fucking awful.
They're supposed to look hastily-made to reflect the imprecision in the estimates. https://www.chrisstucchio.com/blog/2014/why_xkcd_style_graph...
Except that there really isn't imprecision in the estimates. The numbers are really quite precise and the confidence interval is actually very good. Far better than an XKCD style should be used for.
I use XKCD stuff for WAG's, not well-supported data.
Re: Half the papers at NIPS would be rejected if the review process were rerun
#29I think people are reading more into this than there is. Reviewing papers is a highly subjective, high variance process, and very few papers get universally positive reviews.
From the point of view of an author, if you get a paper rejected that you know is worthwhile, you just have to make whatever improvements you can and then submit it again.
Re: Half the papers at NIPS would be rejected if the review process were rerun
#30Something similar happened with IEEE[1] - It had accepted approx 120 papers generated by SCIgen[2]. [1] - http://www.nature.com/news/publishers-withdraw-more-than-120... [2] - http://pdos.csail.mit.edu/scigen/
IEEE may be more prone to precision errors (letting bad papers in) while NIPS may be prone to recall errors (throwing good papers out). With the way reviewing is done (no one can take a week off to read and fully comprehend the four papers they are given) you cannot achieve perfect separation - even if that were possible.
Calling the process of "accepting a SciGen-generated paper into a allegedly peer-reviewed journal" a "precision error" is a bit on the optimistic side. It implies that someone was making a decision after reading the content of the paper, as opposed to, well, just accepting everything in sight.
It doesn't take a "week off" to notice that a paper is gibberish, at the very least.