Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you critically evaluate scientific papers?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Ask HN: How do you critically evaluate scientific papers?

#31

Post by yuan43 »

I think the problem might be with your word "legit." What do you mean by it?

The idea of "legit" does not exist in science. There are hypotheses, created by observation of some phenomenon and an attempt at generalization. There are tests of the hypothesis, which are experiments aimed at disproving it. There's also accumulation of evidence against a hypothesis. Those that don't fall over due to contradictory evidence endure.

Just like nobody rings a bell at the top of a market, nobody blows a horn to announce the arrival of scientific truth. It's all subject to being overturned at any time by new evidence.

When encountering a new paper, this background process is running. Where along the trajectory does this paper lie? Is it contradicting previous findings? If so, how strong is the evidence? How reasonable are the experimental methods? How consistent with the evidence are the conclusions? Was a falsifiable claim ever formulated and/or tested in the first place?

Re: Ask HN: How do you critically evaluate scientific papers?

#32

Post by lapcat »

rat9988 wrote: post #27
lapcat wrote: post #20

I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…

> Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. They go a bit more in depth than that. They try to ensure there is no hole in the state of art, that may have an impact on your result, and that your methodology is sound.

> They go a bit more in depth than that. They try to ensure there is no hole in the state of art, that may have an impact on your result, and that your methodology is sound.

Of course, yes. The reviewers will give critical comments and may even recommend "revise and resubmit" for the authors to make changes to the article.

But the reviewers aren't going to run their own experiments and try to replicate the results, for example.

Re: Ask HN: How do you critically evaluate scientific papers?

#33

Post by 7thaccount »

I read a lot of papers on power systems and tend to look at things in this manner: 1.) Is the author actually trying to solve a real problem or just trying to pad their resume with key words 2.) Is machine learning being suggested even if it is entirely impractical for this use as only slightly slower methods exist that are deterministic and don't require maintenance? 3.) Does the paper make either solid logical arguments or do an experiment on a full-sized model and not just a tiny toy model?

Re: Ask HN: How do you critically evaluate scientific papers?

#34

Post by dahart »

cjbprime wrote: post #24
dahart wrote: post #13

Earlier quoted context omitted.

I kinda agree, but at the same time this seems like an exceedingly pessimistic take. Maybe this can be the devil on one shoulder while you have an angel on the other pointing out that papers already go through peer review, that overt fraud is rare, and that the biggest problem we have isn’t bad science, it’s taking scientific results out of context, misreporting them, overgeneralizing the results, and painting lines…

FWIW, I think you're actually being insufficiently pessimistic, and also the example effects I mentioned (publication bias and multiple hypotheses) exist in the zone where both are true -- the researchers observed something, and also the research claims are false. Peer review is regularly completely inadequate. As an example, consider the amyloid hypothesis for Alzheimer's, which just this week was likely discovered…

Again, while I think those things may be true sometimes, they aren’t the biggest problem we have in terms of science. And I feel like choosing to frame it the way you’re framing it is intentionally ignoring how often things go right, how often people are sincere and honest and competent and right.

We already know peer review isn’t perfect, but it’s also probably a lot better than the layman’s attempt to discriminate good science from bad. We don’t have a ton of data of how many papers are rejected either. But, one high profile example isn’t a particularly scientific way to establish that all of science is actively bad. It’s one bad example, and yes it’s awful that one case can have such wide ranging ramifications, but you’re downplaying the thousands of papers, and thousands of scientists who assumed the research was valid and did good faith work on top of it.

If anything, the practical reaction to your example should be that papers that go high profile need replication studies very quickly, not that we should assume as a lay person that all science is wrong.

Re: Ask HN: How do you critically evaluate scientific papers?

#35

Post by solardev »

Sometimes a lazy shortcut is to read the metastudies instead. That's other researchers aggregating papers on a particular topic and seeing if they statistically agree with each other after normalization of some sort.

You lose a lot of nuance that way, but if you're researching a general topic that's not super niche, it can be a helpful overview.

That's also not for groundbreaking research since it usually takes years, but very little in science is groundbreaking like that anyway, and I doubt a layperson would be able to recognize the true breakthroughs on their own.

Re: Ask HN: How do you critically evaluate scientific papers?

#36

Post by imnotreallynew »

lapcat wrote: post #20

I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…

I’ve long wanted some sort of platform for reading somewhat simplified, summarized research papers on topics of interested. Something a little simpler than an original paper, but more accurate and in-depth than what you might find in a popular news article.

I haven’t found anything loke that yet, and I don’t know if it would be feasible to crowdsource summarized publications. AI summarization isn’t quite good enough yet.

Re: Ask HN: How do you critically evaluate scientific papers?

#37

Post by luispauloml »

ravenstine wrote: post #9
cjbprime wrote: post #5

None of these. Assume any individual paper is wrong. If it's not your current area of research, you can afford to wait for the results to be reproduced (or disproven, or failed to replicate), and read a meta-study instead. Due to a minefield of mistakes like publication bias and multiple hypotheses, not to mention overt fraud, it seems more likely that an exciting result is wrong than that it's correct.

That and don't just read the abstract or conclusion.

And how would I know if a given paper is of interest for me if I don't read the abstract?

Re: Ask HN: How do you critically evaluate scientific papers?

#38

Post by lapcat »

avian wrote: post #26
lapcat wrote: post #20

I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…

> To academicians, the journals are more like "interesting lines of research" rather than "settled results". That may be true around the water cooler. Try publishing any result that goes against a peer-reviewed paper (and, as an author, not be an affilieated with a well-known US university) and you quickly realize that published results are pretty much settled in stone. I'm not talking about ground-breaking stuff. So…

It appears that you're complaining about what gets accepted and what gets rejected for publication to academic journals, which can be a legitimate complaint. The journals are inevitably subject to a form of academic "trendiness" (for lack of a better word).

But that issue is not quite the same as how academics treat journal articles in general. The journals are not a full or fully accurate representation of academia or science as a whole.

Re: Ask HN: How do you critically evaluate scientific papers?

#39

Post by j7ake »

It is difficult to evaluate a single paper without thinking how this paper fits in the broader context and field.

Evaluating a paper involves having spent several years reading many papers and understanding where the field is and where the major challenges are.

Trying to read a single paper in isolation from all the others would lead to over interpretation of the results.

This is why many graduate courses often just involve reading the latest literature and discussing it as a group. Discussing a paper as a group, especially with the presence of a few senior people helps to put the paper in a larger context (because the group together have read more papers than an individual). This greatly accelerated a PhD students understanding of the current state of the art in order to define what are the most interesting next questions.

Overall though, most scientific papers are meant for scientists and the point of them is to figure out whether there is some new idea that can lead to new directions or inquiry. For non practicing scientists, review papers are the way to go because they summarise many papers and try to put them into the context of the open questions.

Re: Ask HN: How do you critically evaluate scientific papers?

#40

Post by thfuran »

imnotreallynew wrote: post #36
lapcat wrote: post #20

I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…

I’ve long wanted some sort of platform for reading somewhat simplified, summarized research papers on topics of interested. Something a little simpler than an original paper, but more accurate and in-depth than what you might find in a popular news article. I haven’t found anything loke that yet, and I don’t know if it would be feasible to crowdsource summarized publications. AI summarization isn’t quite good enough…

Something like Science News?
Post reply on HN