Live data from Hacker News

Half the papers at NIPS would be rejected if the review process were rerun

mrtz.org

51–55 of 55 posts

Re: Half the papers at NIPS would be rejected if the review process were rerun

#51
post #23

For those who didn't read the story: NIPS decided to run this experiment themselves to identify problems. That's a a really brave thing to do, and deserves serious credit.

Absolutely, it's hugely important. The core idea that subjective beings will disagree holds obvious weight, but it takes serious commitment to improving the process to offer up this kind of experiment to prove it.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#53
post #50

For two independent committees, 6% of papers were acceptable without disagreement, 25% were rejectable while the rest were coin flips. This means when your paper gets accepted or rejected, luck is playing huge part. This is not because judges are actually flipping coin but vast majority of people don't seem strikingly good or bad. So for a repeated trial outcome may not be same. Also, the asymmetry here is striking.…

"Why is not every book, t-shirt or smartphone is just great?"

Different people have different ideas of what "great" means. Not everyone thinks the Harry Potter series are great books, while many do. We see that in movies where a movie does poorly at the box office while the critics.

The definition of greatness changes over time, so "It's a Wonderful Life", now considered one of the most critically acclaimed films ever made, had only mediocre revenue when it came out.

Greatness is sometimes situational, so "Dan Brown ... is the undisputed king of airplane books — the not-too-heavy, not-too-long potboilers perfect for a long layover." If you don't fly, then perhaps there's no time when Brown's works might appeal.

Travel has its own category of "good enough." Visiting Germany once I bought a book from the limited English selection not because it was great, but because it was something to read on the long train ride.

A lot of people watch sports, but surely it can't be that all sports games are great, so greatness can't be the only reason for keeping someone's interest.

Since it's hard to predict greatness, people will test out ideas to see if there's a response. Sometimes this can lead to feedback and improvements. Sometimes this testing is through writing clubs. Sometimes (as with smartphone apps) this is with the market itself.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#54
post #53
post #50

For two independent committees, 6% of papers were acceptable without disagreement, 25% were rejectable while the rest were coin flips. This means when your paper gets accepted or rejected, luck is playing huge part. This is not because judges are actually flipping coin but vast majority of people don't seem strikingly good or bad. So for a repeated trial outcome may not be same. Also, the asymmetry here is striking.…

"Why is not every book, t-shirt or smartphone is just great?" Different people have different ideas of what "great" means. Not everyone thinks the Harry Potter series are great books, while many do. We see that in movies where a movie does poorly at the box office while the critics. The definition of greatness changes over time, so "It's a Wonderful Life", now considered one of the most critically acclaimed films eve…

It's simpler than that. We focus on the differences between things, not the similarities. If all movies were equally good, we would then grow to focus on the tiny differences between them and start to judge them based on those.

Re: Half the papers at NIPS would be rejected if the review process were rerun

#55
post #48

Earlier quoted context omitted.

Yes :) My favourite peer review story is when I submitted one on my articles to the top journal in my field at the time (Appied and Enviromental Microbiogy). It came back with the usual peer review trivial changes (cite this irrelevant paper of mine,etc) which I did (this nearly always easier than arguing with the reviewers). The editor made a mistake and instead of sending the updated manuscript out to the original…

They always have to find some way to leave their mark.

This is true, but there is always exceptions. The second paper I published I sent off to the journal and after a couple of months I had not heard anything (this was in the physical paper days where you had to mail everything). My supervisor decided to call the editor to ask what was happening. The editor said "oh we published it last month". The whole paper had gone straight through without a single change. This of course was the last time I ever had a paper accepted like this :)
Post reply on HN