Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you critically evaluate scientific papers?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Ask HN: How do you critically evaluate scientific papers?

#51

Post by pmyteh »

imnotreallynew wrote: post #36
lapcat wrote: post #20

I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…

I’ve long wanted some sort of platform for reading somewhat simplified, summarized research papers on topics of interested. Something a little simpler than an original paper, but more accurate and in-depth than what you might find in a popular news article. I haven’t found anything loke that yet, and I don’t know if it would be feasible to crowdsource summarized publications. AI summarization isn’t quite good enough…

The Conversation[0] tries this. It aims more for 'accurate news stories written by/with academics' than 'simplified research papers' but definitely is trying to be credible yet accessible.

Sadly I've never written for them, as they've not got a huge audience and the career incentives in academia aren't really set up for this kind of accessible popularisation.

[0]: https://www.theconversation.com

Re: Ask HN: How do you critically evaluate scientific papers?

#52

Post by derbOac »

The first step is to think of a result, effect, or phenomenon, not in terms of a paper, but a series of papers or literature.

I almost never give weight to a single empirical paper. I look at them as "hypotheses with data".

If the paper has multiple lines of converging evidence, across multiple samples and methods, it's better.

Replication is better still, and still better with preregistration of some sort.

Really the gold standard to me is meta analysis of multiple studies, trying to account for publication bias. RCTs, large samples, preregistration, multiple labs, skeptics, are all better.

There are degrees of legitimacy, and no silver bullet.

Re: Ask HN: How do you critically evaluate scientific papers?

#53

Post by woojoo666 »

imnotreallynew wrote: post #36
lapcat wrote: post #20

I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…

I’ve long wanted some sort of platform for reading somewhat simplified, summarized research papers on topics of interested. Something a little simpler than an original paper, but more accurate and in-depth than what you might find in a popular news article. I haven’t found anything loke that yet, and I don’t know if it would be feasible to crowdsource summarized publications. AI summarization isn’t quite good enough…

If you're ok with videos, Two Minute Papers[1] is a great one for AI papers

[1]: https://www.youtube.com/channel/UCbfYPyITQ-7l4upoX8nvctg

Re: Ask HN: How do you critically evaluate scientific papers?

#54

Post by gus_massa »

what_is_orcas wrote: post #29
lapcat wrote: post #20

I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…

Yes and no. They're "interesting lines of research" except when they're in your field, then they're foundations or anchor points for your work, past, present, and future. I'm not going to disagree that a lot gets lost in translation between academic article and the lay-person who hears a story on the nightly news, but I don't think gatekeeping academic research is the answer, but rather more emphasis on critical thin…

I guess it depend on the area, but (for me, in my area) published results are just a stating point.

When reading a paper the first question are: Does it even makes sense? (mostly yes, it depends on the journal) Is it overhyped? (definitively yes) Is it interesting after removing the overhype? (sometimes yes) Can we reproduce the result with our tools? (I hope so) Can we combine the result with our previous results? (I hope so) Is the combination interesting? (I hope so) Interesting enough to assign it to a graduate student, to drop whatever we are doing or something in between? (mostly something in between)

If we try to reproduce the result and fail miserably or if the methods are too unclear, we will jut forget about the paper and work in another topic. Perhaps contact the authors if we know them or the result very promising.

I think we never take a result at face value, except perhaps for the list of citations that appear in the introduction about recent related work.

Re: Ask HN: How do you critically evaluate scientific papers?

#55

Post by gus_massa »

avian wrote: post #26
lapcat wrote: post #20

I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…

> To academicians, the journals are more like "interesting lines of research" rather than "settled results". That may be true around the water cooler. Try publishing any result that goes against a peer-reviewed paper (and, as an author, not be an affilieated with a well-known US university) and you quickly realize that published results are pretty much settled in stone. I'm not talking about ground-breaking stuff. So…

It's easier to publish a result that contradicts a previous result but is different, like we measures X=20 but the previous paper said that X=10.

Publishing a negative result is very difficult. I guess you need a long history of good publications to convince the journal you are not making a silly mistake.

Re: Ask HN: How do you critically evaluate scientific papers?

#56

Post by e63f67dd-065b »

dahart wrote: post #13
cjbprime wrote: post #5

None of these. Assume any individual paper is wrong. If it's not your current area of research, you can afford to wait for the results to be reproduced (or disproven, or failed to replicate), and read a meta-study instead. Due to a minefield of mistakes like publication bias and multiple hypotheses, not to mention overt fraud, it seems more likely that an exciting result is wrong than that it's correct.

I kinda agree, but at the same time this seems like an exceedingly pessimistic take. Maybe this can be the devil on one shoulder while you have an angel on the other pointing out that papers already go through peer review, that overt fraud is rare, and that the biggest problem we have isn’t bad science, it’s taking scientific results out of context, misreporting them, overgeneralizing the results, and painting lines…

Overt fraud is rare, but sometimes the line of fraud is worryingly thin. Sure, making up data is fraud, but what about stopping your experiment early? Discarding some perfectly good data points? Tweaking your model until it just falls below p=0.05? These are all real things that are depressingly common in real studies published in prestigious journals. Sketchy correction factors, unsound data collection methodologies, baffling experimental design, the list goes on and on.

Something like a third of all published studies fail to replicate in the worst offending fields, and even more fail to replicate at the original effect size or greater. The incentive structure of the entire institution is screwed. Pessimism is warranted here, if you open a journal you should keep in mind that for most fields there’s a double-digit chance that what you’re reading will fail to replicate at all and even more that it’ll fail to replicate at the same of greater effect size.

Re: Ask HN: How do you critically evaluate scientific papers?

#57

Post by physicsguy »

You often can’t. It depends on how accessible the field is, but in my area, you can’t pick out a single paper and judge it as “good” or “bad”, you need an understanding of the field, etc. etc. and that takes at least months and often a couple of years to build up. If I was to go and move to a new area of Physics, I would seek out the widely cited papers first and spend time making sure I understood the key foundations, probably pick up a recent-ish graduate level text book, etc. etc. before trying to go to modern papers now topic.

Almost all papers will overstate their own importance in the introduction, because you have to do that to get published. It takes context of the field to know for e.g. “oh, this guy is just running random simulations and picking out odd results and publishing them” vs “this person has done a comprehensive set of simulations by doing realistic parameter sweeps and has published the whole results set”.

When reporters write stories, they often just take press releases that groups put out themselves. Some groups sell themselves better than others. In my old field, one group in France looks “dominant” if you look at the news, but in reality they’re one of many that do good work, the others just have less press around it because they’re not working at such press hungry organisations.

Re: Ask HN: How do you critically evaluate scientific papers?

#58

Post by Eddy_Viscosity2 »

imnotreallynew wrote: post #36
lapcat wrote: post #20

I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…

I’ve long wanted some sort of platform for reading somewhat simplified, summarized research papers on topics of interested. Something a little simpler than an original paper, but more accurate and in-depth than what you might find in a popular news article. I haven’t found anything loke that yet, and I don’t know if it would be feasible to crowdsource summarized publications. AI summarization isn’t quite good enough…

I like this idea. When doing a literature review I make summary notes of the paper which I store with it. I use zotero for this which has an add-in (zotfile) that allows me to extract highlights and annotations I make in the pdf to separate notes; this is super handy. So, if those summary notes were somehow made public and organized and if many people did this, they would be a nice source of paper summaries from several perspectives.

Re: Ask HN: How do you critically evaluate scientific papers?

#60

Post by bumby »

ramraj07 wrote: post #42

I want to pass forward words given to me during my masters (by a prof Michel Bouvier if interested) that has stuck with me forever, on how to read a paper: 1. Read the abstract super carefully. Typically nothing in a paper that’s not in the abstract should be considered as absolutely confirmed by that paper. 2. If you’re new to the field, you can read the introduction. Ideally you shouldn’t need to. 3. Directly skip…

I can't help but think this is sometimes an excuse to not read the actual article. If one doesn't have the time to read it, that's understandable. But the best response would be to recuse yourself from the review rather than avoiding reading it. If it's unclear after reading it, it's fine to reject it but I think we owe it to the authors to at least read it in its entirety.

It's not uncommon to have to explain this to a junior researcher when they are confronted by review comments that were clearly already answered in text and the reviewer just didn't read it.[1] It does a disservice to the field and, IMO, is unprofessional.

[1] Before anyone jumps to conclusions like "Those questions should have been clear from the figures", sometime they are questions that are not suitable for being answered in figures. Examples being, "The research should have explained why these particular ML models were used" when the paragraphs directly state things like "These ML methods were chosen because..."

Post reply on HN