What do you evaluate to decide for yourself whether, or not, a scientific paper or experiment is legit?
Ask HN: How do you critically evaluate scientific papers?
1–10 of 63 posts
Re: Ask HN: How do you critically evaluate scientific papers?
#2https://pages.cs.wisc.edu/~akella/CS740/S12/740-Papers/paper...
here are some good tips:
How to Read a Paper
S. Keshav (2016)
David R. Cheriton School of Computer Science, University of Waterloo Waterloo, ON, CanadaRe: Ask HN: How do you critically evaluate scientific papers?
#3I'm by no means an academic, but...
1) is it a journal that I've heard of? Does it have a reputation?
2) who are the authors? what else have the published in the same vein?
3) who funded the research?
4) what sort of content does it survey? who does it cite?
5) how thorough is the study? Is it a low-power study that barely gets a p there's a lot to look at beyond the content. The content itself is (of course) the most important part, but you might not be able to assess it without a doctorate. That is what peer review is for.
None of these are also necessarily on their own to write a paper off. But I'm going to trust something more that appears in the lancet vs some journal I've never heard of before.
Re: Ask HN: How do you critically evaluate scientific papers?
#4Read “Incerto” by Nassim Nicholas Taleb.
Re: Ask HN: How do you critically evaluate scientific papers?
#5None of these. Assume any individual paper is wrong. If it's not your current area of research, you can afford to wait for the results to be reproduced (or disproven, or failed to replicate), and read a meta-study instead. Due to a minefield of mistakes like publication bias and multiple hypotheses, not to mention overt fraud, it seems more likely that an exciting result is wrong than that it's correct.
Re: Ask HN: How do you critically evaluate scientific papers?
#6It depends. In some cases I've implemented the core concept and tested it on my own work. In other cases I wait to see if it's replicated or used often.
When I don't have those options I read carefully, particularly any evaluation methods. If it's machine learning I also know to look for mistakes in hyperparameter tuning.
That said, an alternate approach is to just read the best papers from well respected conferences and venues in your area, rather than cherry picking particular papers.
Re: Ask HN: How do you critically evaluate scientific papers?
#7Most notable resource I am familiar is this paper, which has over 10,000 citations per Google scholar:
“Why Most Published Research Findings Are False”
- https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/pdf/pme...
- https://en.m.wikipedia.org/wiki/Why_Most_Published_Research_...
- https://scholar.google.com/scholar?hl=en&as_sdt=0%2C21&q=Why...
Re: Ask HN: How do you critically evaluate scientific papers?
#8None of these. Assume any individual paper is wrong. If it's not your current area of research, you can afford to wait for the results to be reproduced (or disproven, or failed to replicate), and read a meta-study instead. Due to a minefield of mistakes like publication bias and multiple hypotheses, not to mention overt fraud, it seems more likely that an exciting result is wrong than that it's correct.
Given a paper, how do you find out if it was reproduced, disproven, or failed to replicate?
Re: Ask HN: How do you critically evaluate scientific papers?
#9None of these. Assume any individual paper is wrong. If it's not your current area of research, you can afford to wait for the results to be reproduced (or disproven, or failed to replicate), and read a meta-study instead. Due to a minefield of mistakes like publication bias and multiple hypotheses, not to mention overt fraud, it seems more likely that an exciting result is wrong than that it's correct.
That and don't just read the abstract or conclusion.
Re: Ask HN: How do you critically evaluate scientific papers?
#10I wish papers had a "this finding/conclusion has been replicated by the following people/groups: ..." as mandatory information, and that it was consensus that any non-replicated paper is considered speculative at best. But obviously that is not how things work.