Ask HN: How do you critically evaluate scientific papers?

> I don't think gatekeeping academic research is the answer

Not sure what you mean by this? The journals are inherently gatekeeping academic research via peer review. But the suggestion isn't that the journals be locked away, merely that journals are not a great way for the public to interact with academia. Many academicians do write for the public and are eager to do so, but in different forms than journal articles.

Depends on the field a bit, but my general process involves using a few questions to signal one way or the other.

1. What are the actual claims made by the authors? That is, what do they claim to have found in the study? If you cannot find these, there is a good chance the paper is not particularly useful.

2. For each claim, what were the experiments that led to that decision? Do those logically make sense?

3. Are the data available, either raw or processed? Try reproducing one of their more critical figures. Pay close attention for any jumps in series (e.g. time suddenly goes backwards -> did they splice multiple runs?), dropped data points, or fitting ranges. If the data are not available, consider asking for them. If the code exists, read through it. Does the code do anything weird which was not mentioned in the paper?

4. How do they validate their methods? Do they perform the appropriate control experiments, randomization, blinding, etc? If the methods are so common that validation is understood to be performed (e.g. blanking a UV-Vis spectrum), look at their data to find artifacts that would arise due to improper validation (e.g. a UV-Vis spectrum that is negative).

5. Do they have a clear separation of train and test / exploration and validation phases of their project? If there is no clear attempt to validate the hypothesis for new samples, there is a good chance the idea does not transfer.

I would recommend “news and views” from nature or review articles in general for a broad summary of a field.

'Critical evaluation' in reality may not be possible if you're not already at least fairly familiar with the field, and the kind of techniques used in the research. If you were working in the field and were thinking about trying to replicate the work, or build on it, generally the procedure is something like:

1) Read the abstract, to see if it's something you're interested in and willing to spend some time one. Reading the introduction will often help here as well.

2) Read the materials and methods section and attempt to reconstruct in your head or on a piece of paper exactly what the structure of the research was. If you're not familiar at least a little with the field this may be a rather difficult exercise, particularly if they use techniques you've never heard of. However, without going through this, you can't really do any critical evaluation of the research. This can be the most difficult part; understanding the techniques (and their various pitfalls etc.) may require reading other papers cited in the bibliography.

3) Read the results, and in particular, focus on the figures, which should correlate with the materials and methods section fairly well. Here look for things like error bars, statistics, clarity, etc. Poorly written papers tend to obfuscate more than enlighten. There should be good correspondence between materials and methods and results.

4) Finally, read the discussion. Overblown claims that don't seem supported by the results are not that uncommon, so retain a healthy sense of skepticism. A paper that discusses uncertainties in the results in an upfront manner is typically more reliable.

Thorougly evaluating a paper in this manner is quite a bit of work, and can take hours of effort, even by someone with actual research experience in the field.

Usually, if it was a field that you are not that familiar with (say as an undergraduate student or a new graduate student), you'd start not with research reports, but with published reviews of relatively recent work, which will give you an idea of what's mostly settled and where the controversies are. If you were completely starting from scratch, a well-regarded textbook could be a better option as an introduction.

I now noticed that I read your comment in the opposite order: "just don't read the abstract". I am glad I asked you, though, because I see your point and I agree with you for I have seen this happening before.