You Should Downvote Contrarian Anecdotes

Are people joking when they defend anecdotal stories with "once this anecdotal story proved to be true and the study was faulty"?

The proposition that the article makes is not that anecdotal stories must be false, but that they might influence the reader more than they should. Thus they should not be encouraged.

Let's take an example, since they seem to work best: vaccines.

Vaccines protect you from the risk of contracting particular diseases, some of which are crippling, lethal, or incurable. Plus, most are extremely effective: once you take your shots, you are effectively immune. That's good.

There is a downside however. Sometimes, vaccines have side effects. Most side effects are quite benign, but if you're unlucky enough, they can be crippling, lethal, or incurable. That's bad.

From a medical point of view, vaccines are a net good (let's leave aside logistic considerations, or the effort required to go to the doctor). When you look at the stats, you stand a much better chance at life and health if you take the shot. Even for relatively minor illnesses like the flu.

Now, let's say someone post a heartbreaking comment about how her 9 year old daughter died of a vaccine shot, with all the gory details about the suffering, how she couldn't participate in her school's festival, the size of the coffin… I'm quite sure there are stories of the kind. Given the sheer amount of readers here, maybe one of you will more or less directly relate to that. My apologies to those who do.

Nevertheless, what makes a good story doesn't necessarily make good evidence. When you know of reliable statistics, and you read a contrarian anecdote, you should shift your belief in the direction of the anecdote by a precise amount, which is almost always tiny. What your brain will actually do behind you back however, is shifting your belief by a significant amount, often crossing the "reasonable doubt" line. That's not rational, but that's what will happen. Nameless statistics feel abstract, remote. An anecdote on the other hand feels concrete, real, close. Worse, you can spend far more time reading about the salient anecdotes than learning about the end results of reliable, but boring, scientific studies.

Another example: you don't win the lottery. Period. You don't know of any close family of friend that ever did. But maybe one of you readers do. Maybe that one could comment and say "Hey, but my cousin did win the lottery!". Would that prove me wrong? Not at all. It's just that when the sample size is huge enough, even the tiniest chance can actualize.

A contrarian anecdote may help expose an article being false or misleading in itself. For example: a review written by the manufacturer itself, or astroturfing.

First we must determine the validity of the article existing in itself and any motivations or biases in it. For this, contrarian anecdotes are useful. This article seems to miss this point.

Once we can accept the article at face value, then we may hit the biases described.

I'd say that contrarian anecdotes are good if -- for instance -- the discussion spawned by the anecdote leads to a better understanding of the subject at hand. However, because anecdotes are stories we tend to remember them vividly and they affect our judgment disproportionally. Human brains are very flawed, and it may very well be the case that the negative effect of contrarian anecdotes vastly outweighs the benefit we get from the discussion it spawns. I don't know the answer. But I do know that we can't state with certainty that "having a real discussion is a good thing".

As for how we should weigh new evidence, this is essentially a solved problem: use Bayes' rule. Suppose that 100 out of 100 studies indicate that smoking is a leading cause of cancer and a contrarian viewpoint ("My grandfather smoked his whole life and lived to 123!") indicates otherwise. Then that anecdotal viewpoint should get approximately 0 weight. Zero. Nip. Zilch. Nada.

We're all experts in a few subjects at best. In those subjects we can easily explore different viewpoints, balance different arguments and keep track of the different schools of thought. We can even confidently diverge from expert consensus if needed. But in most subjects we're enthusiastic laymen at best. I don't think debates and exploration of different viewpoints then lead to much greater understanding. Just look at any forum on the internet (including this one). Debates aplenty and the few knowledgeable people get drowned out in a sea of contrarian musings.

Expert consensus is just the aggregate opinion of those who have the best understanding. So when a layman disagrees with the experts he's almost certainly wrong. What I see is the opposite of consensus culture. I see a willingness to disagree with the experts before understanding the subject material in depth.

So you're suggesting that because some people are bad at weighing information, anything based on personal experience should be downvoted? Successful entrepreneurs should no longer give advice? Programmers shouldn't weigh in on programming issues unless they're citing third-party research? Marketers can't suggest tips for how to promote a startup because it's possible their methods aren't universally successful?