Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Half the papers at NIPS would be rejected if the review process were rerun

mrtz.org

Post reply on HN
Print view View original

41–50 of 55 posts

Re: Half the papers at NIPS would be rejected if the review process were rerun

#41

Post by userbinator »

desdiv wrote: post #26
flopto wrote: post #17

Earlier quoted context omitted.

They're supposed to look hastily-made to reflect the imprecision in the estimates. https://www.chrisstucchio.com/blog/2014/why_xkcd_style_graph...

I love the general idea of comic-style graphs, but this particular implementation of it does indeed look "fucking awful" in my opinion. The graph itself is fine, but the axis is made of these faux wavy line that eerily repeats itself.

The graph itself is fine, but the axis is made of these faux wavy line that eerily repeats itself.

I see the same things in fonts that are supposed to look "hand-drawn", and CG renders of realistic scenes - it looks "imperfect" but the way the "imperfection" is itself perfect is what stands out. A little randomness goes a long way to avoiding that.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#43

Post by danieltillett »

IndianAstronaut wrote: post #33
danieltillett wrote: post #19

This is the way "peer review" works. It is basically random. I have always found it comforting whenever I had a paper rejected as I would know it was nothing to do with the quality of my work. I would fix any of the typos found by the reviewers (you always get a spelling nazi as one of reviewers) and send it out again unchanged. I only have had one paper rejected twice and it was accepted unchanged on the third attem…

Isn't this why Mendeley and EndNote exist? Just change up the citation formatting and resubmit to a different journal until one accepts your work.

Yes :)

My favourite peer review story is when I submitted one on my articles to the top journal in my field at the time (Appied and Enviromental Microbiogy). It came back with the usual peer review trivial changes (cite this irrelevant paper of mine,etc) which I did (this nearly always easier than arguing with the reviewers). The editor made a mistake and instead of sending the updated manuscript out to the original reviewers, they sent it out to a new lot of reviewers. What was funny about the whole exercise was the second set of reviewers called the first set of reviewers idiots and told me to change everything back.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#44

Post by drpgq »

andrea_s wrote: post #38

Since NIPS is a very prestigious conference, I'd expect a lot of submitted papers (perhaps a vast majority of them) would fall in a grey area between "clearly unsuitable" and "clearly suitable". I personally think there are too many factors at play in evaluating a series of papers - no objective sorting can really exist. A note for people outside the academic Machine Learning field: NIPS is widely believed to be on a…

CVPR is pretty good.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#45

Post by GFK_of_xmaspast »

userbinator wrote: post #39
onan_barbarian wrote: post #30

Earlier quoted context omitted.

Calling the process of "accepting a SciGen-generated paper into a allegedly peer-reviewed journal" a "precision error" is a bit on the optimistic side. It implies that someone was making a decision after reading the content of the paper, as opposed to, well, just accepting everything in sight. It doesn't take a "week off" to notice that a paper is gibberish, at the very least.

Unless the reviewer doesn't actually know anything at all about what he/she claims to. It wouldn't surprise me at all if most of the general public would be unable to distinguish a SciGen-generated paper from a real one.

What is such a person doing reviewing for the IEEE.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#46

Post by Teodolfo »

techaddict009 wrote: post #21

Something similar happened with IEEE[1] - It had accepted approx 120 papers generated by SCIgen[2]. [1] - http://www.nature.com/news/publishers-withdraw-more-than-120... [2] - http://pdos.csail.mit.edu/scigen/

How is this similar to what the NIPS organizers did? It doesn't seem related at all.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#47

Post by deepsearch »

nl wrote: post #23

For those who didn't read the story: NIPS decided to run this experiment themselves to identify problems. That's a a really brave thing to do, and deserves serious credit.

Exactly, the same credit mimicking cognition along vector comparison deserves.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#48

Post by Natsu »

danieltillett wrote: post #43
IndianAstronaut wrote: post #33

Earlier quoted context omitted.

Isn't this why Mendeley and EndNote exist? Just change up the citation formatting and resubmit to a different journal until one accepts your work.

Yes :) My favourite peer review story is when I submitted one on my articles to the top journal in my field at the time (Appied and Enviromental Microbiogy). It came back with the usual peer review trivial changes (cite this irrelevant paper of mine,etc) which I did (this nearly always easier than arguing with the reviewers). The editor made a mistake and instead of sending the updated manuscript out to the original…

They always have to find some way to leave their mark.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#49

Post by Canada »

Selecting talks really subjective. It's more like choosing what songs to play at a party than an objective sorting process.

You have limited speaking slots. You have to guess what the conference attendees will find interesting this year. You are biased by your own particular interests.

Also, some of the submitters are your friends/colleagues, and even if they didn't tell you what they were submitting already, which is unlikely since your relationship is based on talking about this stuff, you can tell it's theirs in less than 250 words...

However a conference tries to sell the fairness and objectivity of its process, you can't anonymize or double blind these things away.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#50

Post by sytelus »

For two independent committees, 6% of papers were acceptable without disagreement, 25% were rejectable while the rest were coin flips. This means when your paper gets accepted or rejected, luck is playing huge part. This is not because judges are actually flipping coin but vast majority of people don't seem strikingly good or bad. So for a repeated trial outcome may not be same. Also, the asymmetry here is striking. Definitely bad papers dominates in number by 4X than definitely good papers.

These are really great observations with deep implications. This same patterns might get applied in other aspects of life such as interviewing candidates or selection of mate or buying a shirt. In all these cases, we might have similar distribution at work.

I have often wondered why is it so hard to have less mediocrity in world? Why is not every book, t-shirt or smartphone is just great? One obvious reason is that lot of times people create something out of obligation such as demand from job instead of out of urge to create. So subsequent question is that if it was possible that no one has to have any obligation to create, can above distribution turn its head over hill? For example, in that scenario would we have, say, 70% great papers, 5% mediocre and rest coin toss?

Post reply on HN