Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Half the papers at NIPS would be rejected if the review process were rerun

mrtz.org

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Half the papers at NIPS would be rejected if the review process were rerun

#31

Post by army »

dhm wrote: post #22
robotresearcher wrote: post #20

Earlier quoted context omitted.

What does "poor quality" mean? There is no absolute standard for quality. "Poor" is something like "less good than usual compared to the recent work in this community". So the top-scoring third-ish of papers sent to the currently-converged-on favourite venue of a community are pretty much by definition not poor. Unless something very weird indeed happens one year. There are usually only very few really excellent pape…

Ok, we've gone from "top tier venue, basically impossible to have a large fraction of poor papers submitted" to "Most papers are filler in retrospect" and "conferences need to accept a decent number of papers so that people will show up and cover the costs of the meeting". I guess if I am deciding whether or not to hire a professor I would be tempted to disregard publications in this conference.

I think its a bit harsh to call the papers "filler", but the reality is that most papers (in CS, anyway) are incremental work on important but well-studied problems or work on problems that are fairly narrow or not universally considered to be important. Reviewers tend to have wildly divergent opinions on how important or interesting that kind of work is.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#32

Post by liquidise »

dhm wrote: post #15

I am surprised the committees were "tasked with a 22.5% acceptance rate". Couldn't more than 77.5% of the submissions have been of poor quality?

I am more curious of the inverse: what if more that 22.5% of papers were at acceptable quality levels? Wouldn't that leave each committee to pick and choose, thus artificially inflating their disagreements?

Re: Half the papers at NIPS would be rejected if the review process were rerun

#33

Post by IndianAstronaut »

danieltillett wrote: post #19

This is the way "peer review" works. It is basically random. I have always found it comforting whenever I had a paper rejected as I would know it was nothing to do with the quality of my work. I would fix any of the typos found by the reviewers (you always get a spelling nazi as one of reviewers) and send it out again unchanged. I only have had one paper rejected twice and it was accepted unchanged on the third attem…

Isn't this why Mendeley and EndNote exist? Just change up the citation formatting and resubmit to a different journal until one accepts your work.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#34

Post by robotresearcher »

dhm wrote: post #22
robotresearcher wrote: post #20

Earlier quoted context omitted.

What does "poor quality" mean? There is no absolute standard for quality. "Poor" is something like "less good than usual compared to the recent work in this community". So the top-scoring third-ish of papers sent to the currently-converged-on favourite venue of a community are pretty much by definition not poor. Unless something very weird indeed happens one year. There are usually only very few really excellent pape…

Ok, we've gone from "top tier venue, basically impossible to have a large fraction of poor papers submitted" to "Most papers are filler in retrospect" and "conferences need to accept a decent number of papers so that people will show up and cover the costs of the meeting". I guess if I am deciding whether or not to hire a professor I would be tempted to disregard publications in this conference.

> I would be tempted to disregard publications in this conference.

That was not something I suggested. NIPS is very good conference, and a paper there is suggestive of quality work. Lots of past NIPS authors have been aqui-hired or regular-hired by Google and Facebook recently in their machine learning spending sprees, for example.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#35

Post by davmre »

dhm wrote: post #15

I am surprised the committees were "tasked with a 22.5% acceptance rate". Couldn't more than 77.5% of the submissions have been of poor quality?

NIPS gets a ton of submissions, so the law of large numbers governs pretty strongly. Imagine that each paper submitted is independently either good or bad, with 22.5% probability of being good. With 1660 submissions, the total number of good papers follows a Binomial(1660, 0.225) distribution, which has mean 374 and standard deviation 17. Under this model, the fraction of good papers would be somewhere in the range 20.5-24.5% (corresponding to a two-standard-deviation window around the mean) in 95% of reviewing cycles. So even though the quality of the individual papers is totally random, the randomness mostly "cancels out" and the overall number of good submissions is relatively constant.

Of course this is assuming an objective standard for what constitutes a "good" paper. As others have pointed out, the only really meaningful standard is "how does this paper compare to other work being done in this field"? So it's also reasonable to think of NIPS's goal as just trying to present the best papers that were written in any given year, not as bestowing a strictly-defined stamp of objective quality.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#36

Post by wsxcde »

dhm wrote: post #18
wsxcde wrote: post #16

Earlier quoted context omitted.

Highly (almost vanishingly) unlikely at a "top-tier" venue like NIPS.

Can you say more about why you believe this is true?

Consider the demographic submitting to NIPS. It's a self-selected group within the top researchers in the world in that area. The best people in the field don't want to be seen publishing the so called "second-tier" conferences, so they will submit exclusively to the likes of NIPS. And if you're an up and coming researcher or research group, you will want to establish credibility by publishing in these sorts of venues, and you will almost surely send your best work there. Add to this the fact that this is a "hot" field, so more and more researchers and research groups are getting into the field and trying to publish papers, I think it's very likely that NIPS gets a lot more good papers than they can possibly accept.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#37

Post by robotresearcher »

army wrote: post #31
dhm wrote: post #22

Earlier quoted context omitted.

Ok, we've gone from "top tier venue, basically impossible to have a large fraction of poor papers submitted" to "Most papers are filler in retrospect" and "conferences need to accept a decent number of papers so that people will show up and cover the costs of the meeting". I guess if I am deciding whether or not to hire a professor I would be tempted to disregard publications in this conference.

I think its a bit harsh to call the papers "filler", but the reality is that most papers (in CS, anyway) are incremental work on important but well-studied problems or work on problems that are fairly narrow or not universally considered to be important. Reviewers tend to have wildly divergent opinions on how important or interesting that kind of work is.

The "in retrospect" was an important part of that point. Reviewers don't have access to it when reviewing.

Some conferences and journals have a retrospective prize for the best paper of, say, ten years ago. It's a neat way to recognize papers that turned out to be useful.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#38

Post by andrea_s »

Since NIPS is a very prestigious conference, I'd expect a lot of submitted papers (perhaps a vast majority of them) would fall in a grey area between "clearly unsuitable" and "clearly suitable". I personally think there are too many factors at play in evaluating a series of papers - no objective sorting can really exist.

A note for people outside the academic Machine Learning field: NIPS is widely believed to be on a different level from the rest - during my PhD, my advisor used to say that a NIPS paper would be on par with a paper published on a good journal, resume-wise. The difference is especially striking if you've had the chance to attend other conferences (including, alas, IEEE-sponsored events) - that are, with very few exceptions, fairly terrible from a scientific point of view.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#39

Post by userbinator »

onan_barbarian wrote: post #30
sqrt17 wrote: post #25

Earlier quoted context omitted.

IEEE may be more prone to precision errors (letting bad papers in) while NIPS may be prone to recall errors (throwing good papers out). With the way reviewing is done (no one can take a week off to read and fully comprehend the four papers they are given) you cannot achieve perfect separation - even if that were possible.

Calling the process of "accepting a SciGen-generated paper into a allegedly peer-reviewed journal" a "precision error" is a bit on the optimistic side. It implies that someone was making a decision after reading the content of the paper, as opposed to, well, just accepting everything in sight. It doesn't take a "week off" to notice that a paper is gibberish, at the very least.

Unless the reviewer doesn't actually know anything at all about what he/she claims to.

It wouldn't surprise me at all if most of the general public would be unable to distinguish a SciGen-generated paper from a real one.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#40

Post by wsxcde »

liquidise wrote: post #32
dhm wrote: post #15

I am surprised the committees were "tasked with a 22.5% acceptance rate". Couldn't more than 77.5% of the submissions have been of poor quality?

I am more curious of the inverse: what if more that 22.5% of papers were at acceptable quality levels? Wouldn't that leave each committee to pick and choose, thus artificially inflating their disagreements?

Yes, and I think that is essentially why we're seeing these disagreements.

I've heard from lots of professors that a good conference gets a lot of "very-good-but-not-great" submissions and the job of the program committee is to pick the best among these. I wouldn't be surprised at all if minor personal preferences (which from the outside look rather random) ended up having a big say in the fate of a particular paper. Maybe some reviewers are more forgiving of poorly-written but technically strong papers, maybe some reviews consider certain fields "dead" and so are biased against them, reviewers tend to wildly different standards on how extensive an experimental analysis should be to be acceptable, ...

Post reply on HN