[deleted]
Ask HN: How do you critically evaluate scientific papers?
11–20 of 63 posts
Re: Ask HN: How do you critically evaluate scientific papers?
#12This is not easy to do, even for experts. An expert will often take several days to do a peer review of a paper. It's not really possible to review papers outside a narrow field of expertise, partly because doing so requires being well-read in the relevant literature, and knowing which groups are working on what topics, what methodologies are showing promise, and so forth.
And peer review does not get to the question of whether things are "legit". Likely you are quite aware of problem cases that come up in the news every once in a while. Bear in mind that those problem cases were peer reviewed, often through many, many stages, and nobody twigged to the problem.
Perhaps the answer to "what" question is: get an advanced degree, work for a decade or so in a particular sub-field, attend conferences to know what is being done at the cutting edge, spend a lot of time reading, and use all the tools at your disposal to evaluate the work.
In a nutshell, I'd say that non-experts should not expect to be able to evaluate the legitimacy of scientific papers and experiments. And even a highly-skilled expert in one sub-field is a non-expert in another.
There is no simple N-step program to yield what seems to be sought in your question.
Re: Ask HN: How do you critically evaluate scientific papers?
#13None of these. Assume any individual paper is wrong. If it's not your current area of research, you can afford to wait for the results to be reproduced (or disproven, or failed to replicate), and read a meta-study instead. Due to a minefield of mistakes like publication bias and multiple hypotheses, not to mention overt fraud, it seems more likely that an exciting result is wrong than that it's correct.
I kinda agree, but at the same time this seems like an exceedingly pessimistic take. Maybe this can be the devil on one shoulder while you have an angel on the other pointing out that papers already go through peer review, that overt fraud is rare, and that the biggest problem we have isn’t bad science, it’s taking scientific results out of context, misreporting them, overgeneralizing the results, and painting lines where none exist. The actual observations in a paper are usually real and legit, the problem is that humans seek to explain those observations as a pattern, and describing that pattern and assuming it even exists, that’s where we fail so often. More often than the paper’s conclusion being wrong, people cite a paper as evidence for something “related” or “similar”, where the paper’s observations don’t actually apply.
Re: Ask HN: How do you critically evaluate scientific papers?
#14If you mean "legit" as opposed to using false data, well, you can't. That is one of the issues that replication solves, so you can look for replication, but it's missing for most papers, whatever their quality.
If you mean "legit" as in a high quality work that you can trust the conclusions, the things that I see people get wrong most of the time are:
- Make sure what you understand the conclusion is is exactly the same thing the paper concludes. That is the one I see people doing wrong most often, if you are not an academic, usually papers don't say what you think they say. One of the things to do here is taking your list of fallacies, and looking if you have fallen for any of them.
- Make sure the paper's conclusion is supported by its body. Yep, once in a while the above problem affects the scientists themselves, not only outsiders. And peer reviewers are not immune to it either.
- Take into account the paper's (predictive or explanatory) power. For more complex experiments, it's summarized as the p-value. Keep in mind that there may be many papers out there not published because they got the boring result, so the odds that you are looking at an statistical oddity is higher than it seems. It's usually good to ignore a single paper with low p-values (like 0.95, but if it's a popular research topic, maybe even 0.995) and wait for a trend to appear between papers. But also here, try to answer how the hypothesis would apply to different populations, and if there is any reason the paper got a biased result.
- If you can, look at how any experiments were conducted, and if (and how) the author corrected for confounding elements. But this one already requires a lot of know-how.
Re: Ask HN: How do you critically evaluate scientific papers?
#15Most notable resource I am familiar is this paper, which has over 10,000 citations per Google scholar: “Why Most Published Research Findings Are False” - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1182327/pdf/pme... - https://en.m.wikipedia.org/wiki/Why_Most_Published_Research_... - https://scholar.google.com/scholar?hl=en&as_sdt=0%2C21&q=Why...
This does not help evaluate a paper.
Plus, there's only a very few fields that come down to a single p-value like this setup.
Re: Ask HN: How do you critically evaluate scientific papers?
#16I wish papers had a "this finding/conclusion has been replicated by the following people/groups: ..." as mandatory information, and that it was consensus that any non-replicated paper is considered speculative at best. But obviously that is not how things work.
[deleted]
Re: Ask HN: How do you critically evaluate scientific papers?
#17I think it's a process that's best learned by watching other scientists do it. That's how I've learned to do so. Though you're never done -- you can always improve your bullshit detectors.
There's always obvious things to catch: do all of the data points plotted on this graph make sense? Does a predictive model fall back to chance performance when it should? Should the authors need to z-score in order to see this effect? Other labs typically use X statistical model here: why are these people using something different, and how should I expect the results to change? If my brain is having a difficult time understanding how they went from raw data to a given plot, that's usually an indication that something's fishy.
Then you need to check for consistency between the results. Figures should come together to tell a coherent story. If there are small discrepancies in what Figure 1 and Figure 3 are telling you, investigate. It could be that the authors are hiding something.
Then there's sociological factors: how much do I trust the work that this group puts out? Being in higher-prestige journals is actually an indication that you should be more skeptical of the results, not less. There's more riding on the results telling a neat and coherent story, and more space to hide things in (given that you typically need to do 2 or 3 papers worth of experiments to publish in Nature these days).
Unfortunately, there's a great deal of bullshit published every day. Every paper has a little bit of it. The trick is to get great at spotting it and getting something out of the paper without getting discouaged with science as a whole.
Re: Ask HN: How do you critically evaluate scientific papers?
#18I wish papers had a "this finding/conclusion has been replicated by the following people/groups: ..." as mandatory information, and that it was consensus that any non-replicated paper is considered speculative at best. But obviously that is not how things work.
That's not the purpose of papers, the purpose of papers is to communicate to other scientists that might attempt to replicate the results.
People want "here's solved science" as fast as possible, which is understandable! We all do! But there has to be some channel to first disseminate results, and ideally that's a channel that is open to all, and not hidden from the public and limited to scientists only.
That channel is scientific publication. And science is not error-free, it is self-correcting. So expect errors, and expect them to hopefully be identified.
If you want things that are 100% solved, go to the science from engineering textbooks. If you want to be on the cutting edge, there's always some risk that the frontier of discovery will be wrong.
Re: Ask HN: How do you critically evaluate scientific papers?
#19None of these. Assume any individual paper is wrong. If it's not your current area of research, you can afford to wait for the results to be reproduced (or disproven, or failed to replicate), and read a meta-study instead. Due to a minefield of mistakes like publication bias and multiple hypotheses, not to mention overt fraud, it seems more likely that an exciting result is wrong than that it's correct.
Given a paper, how do you find out if it was reproduced, disproven, or failed to replicate?
Google “replication crisis” — its huge topic, lots of information.
Re: Ask HN: How do you critically evaluate scientific papers?
#20I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results".
Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not imply that the reviewer endorses any conclusions or arguments made by the article.
It's usually not a great idea for laymen to be reading academic journal articles, and it's especially not a great idea for the news media to report them. It's not that academia is trying to "hide" things from the public; rather, the academic journals are simply not writing for a public audience, otherwise they'd write differently.
Sadly, there's also the "publish or perish" factor: as an academic, you have to publish something to justify your existence in academia, get tenure, etc. So there's a lot of material that in an ideal world might be considered filler.