Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Peer review: the end of an error?

the-tls.co.uk

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Peer review: the end of an error?

#71

Post by gowld »

noobermin wrote: post #48
jhbadger wrote: post #44

I think the mathematical sciences see peer review as less of a requirement than the experimental -- in theory, anyone can verify a proof for themselves. But in experimental science you really need someone familiar with the techniques to make sure that they did all the needed controls and didn't do any shortcuts and that's what peer review is useful for.

Was going to comment to say the exact same thing. OP used a paper by medical professionals to poo-poo formal peer review and then used math papers as examples of successful arxiv papers scrutinized by other scientists. After realizing that, I stopped reading the article then Googled his name, and--surprise, surprise--he's a mathematician. It will work for mathy and theoretical sciences like CS and theoretical physics…

It sounds like you're saying that experimentalists rely on peer review as a way to fraudulently cover up their shoddy experimental methods.

Re: Peer review: the end of an error?

#72

Post by gowld »

improbable22 wrote: post #42
aldoushuxley001 wrote: post #25

Earlier quoted context omitted.

"Only interesting problems/solutions/results get published" I'm afraid we're already there. Journals are very much strongly incentivized to publish only interesting or novel results.

Isn't this the point of journals? They communicate new knowledge, which changes the status of what is known. There's a balance to be struck in how certain these things should be before being published, for sure. The p-hacking plague is a symptom of setting this certainty bar too low, or equivalently, the required level of novelty too high. Things which stay constant are the job of standards bodies, and textbook write…

http://www.jnr-eeb.org/index.php/jnr

Re: Peer review: the end of an error?

#73

Post by isolli »

John_KZ wrote: post #18
woliveirajr wrote: post #3

> Remarkably, we have arrived at a system where academics feel a moral obligation to perform the thankless task of reviewing the work of other academics, anonymously and unpaid. This undoubtedly makes the literature better than it would otherwise have been, and ensures at least one reader for each paper. I'm afraid it doesn't ensure that all papers will be read. Some papers can attract more attention, some papers can…

Exactly. It's the same fallacy with open source code. Just because it could be reviewed, doesn't mean it has been. Formal or semi-formal groups that professionally review papers are essential. How are they going to get paid? If you want research to be publicly available (as it should be) you can either put the reviewers on government dime (which isn't a great idea but it's better than nothing) or you can follow the s…

Pull-request reviews work the same as peer review: it ensures that at least one person (the reviewer) reads the code or article.

Re: Peer review: the end of an error?

#74

Post by noobermin »

gophile wrote: post #67
noobermin wrote: post #49

Earlier quoted context omitted.

If anything, that works as a great defense of peer review given Einstein was wrong.

Einstein was right about some things and wrong about some things. Which work of Einstein are you referring to when you imply that a peer review could have detected the error in his work?

The story in the linked blogpost.

Re: Peer review: the end of an error?

#75

Post by noobermin »

gowld wrote: post #71
noobermin wrote: post #48

Earlier quoted context omitted.

Was going to comment to say the exact same thing. OP used a paper by medical professionals to poo-poo formal peer review and then used math papers as examples of successful arxiv papers scrutinized by other scientists. After realizing that, I stopped reading the article then Googled his name, and--surprise, surprise--he's a mathematician. It will work for mathy and theoretical sciences like CS and theoretical physics…

It sounds like you're saying that experimentalists rely on peer review as a way to fraudulently cover up their shoddy experimental methods.

Hmm, if you're lab A, why would you help to do the cover up of lab B's work, especially if you might be competing with them?

Scientists at least in my field don't do each others' dirty work, we're much more disconected from each other than that. Also, peer review is anonymous.

Re: Peer review: the end of an error?

#76

Post by tom_mellior »

wyattpeak wrote: post #60
danieltillett wrote: post #37

The big problem with peer review is almost no reviewing takes place and it is at best is either a way for your enemies to get revenge or a cheap proofreading service. The solution is to offer a largish reward to the most constructive peer review for each manuscript chosen by the editor, or better by other reviewers. If I (or more likely one of my starving grad students) knew that I would get $1000 for providing the m…

But who decides what's constructive? It's all very well if you're choosing amongst people who basically agree with you - what if they're telling you it's going to take another year of experiments to get it up to standard? Which reviewers, too, do you respond to? If it's a popular paper you probably can't respond to all criticism, but picking which criticism to interact with isn't going to lead to good science. There'…

> But who decides what's constructive?

For a start, it would be good if as an author I could give feedback on review quality. Rebuttal periods are becoming standard for computer science conferences, and you can put some feedback into your rebuttal, but if you are too frank about bad reviewers, they could decide to recommend rejection. It would be good to add an option to review reviewers after the final decision on the paper has been made. I do think that that would make (some) reviewers more constructive.

A few conferences do have distinguished reviewer awards, though I've never really seen explained how the winners are chosen.

Re: Peer review: the end of an error?

#77

Post by lowkeyokay »

I really think we should have something like Stack Overflow for academic papers. The comments system works so well. Also, the so called impact factor of journals, which in many cases are directly or indirectly linked with academics and universities financial outcome, could be replaced by the points system from SO.

Of course I'm not proposing using SO directly for academic research but something inspired by it.

Re: Peer review: the end of an error?

#78

Post by tom_mellior »

xamuel wrote: post #33
danieltillett wrote: post #30

Earlier quoted context omitted.

Didn’t your supervisor step in and say this is not up to scratch? When I was a supervisor a good chunk of my time was spent going through my student’s and postdoc’s manuscripts to ensure nothing was in them that might embarrass us both.

I was reckless and wrote papers unrelated to my dissertation and submitted without going through my advisor. Fortunately, the bad ones were rejected. The good ones ended up making me well-recognized in some extremely tiny niches, and ultimately did way more for my career than my PhD work did :D I'm really glad some of the bad ones didn't see the light of day :P

> I was reckless and wrote papers unrelated to my dissertation and submitted without going through my advisor.

This is the opposite of reckless, this is how you make a name for yourself, and how you prove that you can develop an independent research agenda. As you say, it's good for your career, and it's good for science too (if the work is good, of course). Good for you!

Re: Peer review: the end of an error?

#79

Post by hvidgaard »

lowkeyokay wrote: post #77

I really think we should have something like Stack Overflow for academic papers. The comments system works so well. Also, the so called impact factor of journals, which in many cases are directly or indirectly linked with academics and universities financial outcome, could be replaced by the points system from SO. Of course I'm not proposing using SO directly for academic research but something inspired by it.

I think a hybrid between SO and Discourse for papers would be more fitting. There really needs to be an open but moderated discussion for the papers - that'll serve as the peer review.

Re: Peer review: the end of an error?

#80

Post by tincholio »

pizlonator wrote: post #57
tincholio wrote: post #53

Earlier quoted context omitted.

The difference is that more often than not, you can't go to the same level of detail and depth in a blog post as you would in a scientific article (and if you do, not that many people will read it). Reading science is hard work, and properly reviewing a "journal length" paper can take upwards of a day. I don't think you can have it both ways. If you write a good vulgarization, or technical blog post, and submit it he…

My blog posts typically have more scientific content than a CS paper.

I'd be happy to see some, please link
Post reply on HN