Peer Review is No Panacea
theatlantic.com
Peer Review is No Panacea
1–10 of 22 posts
Re: Peer Review is No Panacea
#2"There are No Panaceas" meme: panacea for "[X] is No Panacea" headlines?
Re: Peer Review is No Panacea
#3This is much like unit-testing: it can prove the presence of bugs but not the absence. Still, it's necessary, despite its shortcomings.
Re: Peer Review is No Panacea
#4"There are No Panaceas" meme: panacea for "[X] is No Panacea" headlines?
Meme templates considered harmful.
Re: Peer Review is No Panacea
#5Summary: Imperfect system is imperfect.
'Ya gotta love a tautology adorned as journalism.
Are there any reasonable alternatives for privately funded research? No. There will always be the urge to game data to fit the hypothesis and honest people will still make mistakes. This forever shall remain a challenge for science.
I'm struggling with the urge to flag, but recognize the applicability to HN.
Re: Peer Review is No Panacea
#6Summary: Imperfect system is imperfect. 'Ya gotta love a tautology adorned as journalism. Are there any reasonable alternatives for privately funded research? No. There will always be the urge to game data to fit the hypothesis and honest people will still make mistakes. This forever shall remain a challenge for science. I'm struggling with the urge to flag, but recognize the applicability to HN.
[deleted]
Re: Peer Review is No Panacea
#7Summary: Imperfect system is imperfect. 'Ya gotta love a tautology adorned as journalism. Are there any reasonable alternatives for privately funded research? No. There will always be the urge to game data to fit the hypothesis and honest people will still make mistakes. This forever shall remain a challenge for science. I'm struggling with the urge to flag, but recognize the applicability to HN.
I fail to see your point. The core claim that there are people who put far too much stock in the words-treated-as-magic "peer review" is definitely true; I encounter them all the time, to the point that I've also come to recognize it as a pattern. The fact that this is not true is, as you say, tautological. But some people need this fact brought to their attention, as it is clearly not generally understood.
I've come to classify attitudes about science now into three broad categories. Hostilty: "I don't care what science says, Homeopathy worked for me." Religious: "Peer review is never wrong, and therefore the consensus is also always correct in every particular and those who disagree are heretics of science!" Realistic: "Science is the best way we have to find truth, but it isn't perfect either. The history of science should be learned and carefully examined, and it should not be forgotten that everything we see in the past is still around today, the only question is where." Rather a lot of people fit into that second one, and think it is the best understanding of science there is, and tend to label people with the third level of understanding as heretics on par with homeopaths, too. Trying to prod people out of religion-of-science into realistic-understanding-of-science is a noble goal, worthy of a blog post.
Science has been wrong before, for decades at a time in every discipline I've ever studied enough to learn about its history (in addition to the current consensus), and the question is not whether there are disciplines every bit as wrong right now, but which ones they are. If you don't understand that or haven't internalized it, you don't understand the process of science, even if you can recite its tenants at will. When we say that science is a process, not an end-result, that isn't just words, it is the reality, and what it means in practice is that the consensus is very frequently wrong. There's nothing to "correct" in consensus if it's presumed always correct. The religious idea of science makes no sense; consensus must be wrong at some points in time, it is a necessary step.
(This is all discipline-neutral, by the way. I do have my opinions about which disciplines are most likely to be wrong right now, but I freely acknowledge those are opinions. Certainly in the past entire disciplines have made glaringly-obvious-in-hindsight-and-common-sense mistakes before and it is no great insight to think that there might still be such errors around. But again, I freely admit that only time will really tell, and there are certainly other wrong-consensus-views I agree with.)
Re: Peer Review is No Panacea
#8In my personal experience, peer review follows fashion, and fashion is generational. Couch your results in the fashion of the day, cross your t's, dot your i's, and you're all set. Publish frequently, in innocuous little bites. Wait for the old guard to retire before publishing the synthesis that brings your prior publications together into a new and coherent whole.
Now you're the old guard. Repeat.
Re: Peer Review is No Panacea
#9I would Love it if referee reports were published with papers. Why waste all of the unpaid time and research of the referees by compressing their opinions into a yes/no/revise decision? This would improve the quality of the reports, and make the process of reading papers much easier. One of the most difficult parts of reading a paper is trying to decide for yourself if the methods or conclusions of the paper are flawed. The referee reports are a valuable counterpoint from people who probably know more about a topic than you do!
Re: Peer Review is No Panacea
#10Summary: Imperfect system is imperfect. 'Ya gotta love a tautology adorned as journalism. Are there any reasonable alternatives for privately funded research? No. There will always be the urge to game data to fit the hypothesis and honest people will still make mistakes. This forever shall remain a challenge for science. I'm struggling with the urge to flag, but recognize the applicability to HN.
I fail to see your point. The core claim that there are people who put far too much stock in the words-treated-as-magic "peer review" is definitely true; I encounter them all the time, to the point that I've also come to recognize it as a pattern. The fact that this is not true is, as you say, tautological. But some people need this fact brought to their attention, as it is clearly not generally understood. I've come…
I also think a big part is how forthright a scientist is. Most scientists claim way more than can be supported by their evidence, and it's not always clear what assumptions are being made. There's a great talk, I can try to find it if there is interest, that discuss the hubris that most climate scientists have. Observing a system for a tiny portion of a cycle can in no way tell you the entire cycle. If you then admit that "hey we don't know, but here's our best guess, and here's why we think it" then you've got something. Once something is established like like copper conducts electricity or the speed of light then you can say "hey we're pretty sure about this". It has a lot to do with the maturity of the science and the tools and techniques available to the scientists. Complex systems that cannot be decomposed and upon which experiments can not be made have should have a much lower confidence value than repeatable tried and true experiments. To conflate them is to cause confusion and to weaken the trust in more established science. Peer review is more about how to position your work in relation to others for ease of understanding than to verify the results independently.