21–30 of 63 posts
Earlier quoted context omitted.
Given a paper, how do you find out if it was reproduced, disproven, or failed to replicate?
Google “replication crisis” — its huge topic, lots of information.
Also, I do recall a book Bad Science that I've had on my read list for a while now ;) It seems to cover the topic but since I didn't read it I can't vouch for it.
Besides some of the suggestions already presented, here are some additional elements I look for:
1) Do the references actually align to there claims they back up? Surprisingly, you can find a lot of reference fodder that is misused
2) Does the validation make sense or is it gamed? There are times the article method uses one model but slightly tweaks it during validation. Examples being dropping troublesome data or using a different classification scheme. Does the model actual compare to and improve on current state-of-the-art practice?
3) Do the authors offer their data up for review?
None of these. Assume any individual paper is wrong. If it's not your current area of research, you can afford to wait for the results to be reproduced (or disproven, or failed to replicate), and read a meta-study instead. Due to a minefield of mistakes like publication bias and multiple hypotheses, not to mention overt fraud, it seems more likely that an exciting result is wrong than that it's correct.
I kinda agree, but at the same time this seems like an exceedingly pessimistic take. Maybe this can be the devil on one shoulder while you have an angel on the other pointing out that papers already go through peer review, that overt fraud is rare, and that the biggest problem we have isn’t bad science, it’s taking scientific results out of context, misreporting them, overgeneralizing the results, and painting lines…
Peer review is regularly completely inadequate. As an example, consider the amyloid hypothesis for Alzheimer's, which just this week was likely discovered to be actively fraudulent after nearly 20 years, thousands of papers, and thousands of scientists spending their entire research careers on it.
I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…
That may be true around the water cooler. Try publishing any result that goes against a peer-reviewed paper (and, as an author, not be an affilieated with a well-known US university) and you quickly realize that published results are pretty much settled in stone.
I'm not talking about ground-breaking stuff. Something on the level of "after lots of careful work in our group we were unable to replicate the results published in the relatively well-cited paper in a very niche topic" is largely unpublishable.
I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…
They go a bit more in depth than that. They try to ensure there is no hole in the state of art, that may have an impact on your result, and that your methodology is sound.
Look especially for meta analysis, where other scientists have gone through and critically evaluated a result, aggregated similar experiments, and checked publication bias.
I think a lot of people outside of academia misunderstand the role of academic journals. To academicians, the journals are more like "interesting lines of research" rather than "settled results". Likewise, "peer review" typically just means that 3 academics read the article and thought it was interesting and worth publishing. It's important to recognize that a reviewer accepting an article for publication does not im…
I'm not going to disagree that a lot gets lost in translation between academic article and the lay-person who hears a story on the nightly news, but I don't think gatekeeping academic research is the answer, but rather more emphasis on critical thinking and critical reading.
Usually on skimming and first read I'll flush out all the concepts and words I don't understand. I can then figure those out when reading about the paper. Reading what other people have written about it will help you understand more as well. Then, on the final reread I'm prepared to understand.
At this point, I'm either satisfied with some takeaway (usually much narrower than any press about the article) or I have doubts or questions. Search to see what others say about the doubts or questions. Then, read references around central or implausible claims. It's not really uncommon to find references that are dubious - e.g. the author cites a figure from another paper, you go to that paper and it gets the figure from somewhere that gets it from somewhere that gets it from someone's example of a different thing.