Fighting Brandolini's Law with Sampling

Most everything is "considered biased" by some people. In this case, Zuckerberg and the Bain employee who authored that report are indeed people -- 2 out of billions.

Consider an alternative framing, "fact checkers like Politifact are precisely what are considered UNbiased". It is at least as true (because at least 2 people consider it to be so).

Given that framing alternative to yours: what, if anything, should we do anything about the situation?

How do you think framing, rather than substance, affects that discussion?

Both this and the underlying system of fact checking are ignoring the elephant in the room: we have no direct access to the truth. Instead, all we can do is check for consistency. This can be either internal (if I say "two is even" and later "two is odd" I must have lied at least once) or with external source (e.g. look it up somewhere, or ask an expert).

The best external source is reality, if you can corner it with a well designed experiment; this is, unfortunately, really, really hard.

Established theories are also good (but, as history has shown, can be wrong). The biggest problem with theory-based fact checking is that our best theories generally come in pairs that make conflicting claims or are otherwise inconsistent. Plus, the proper application of theories can often be a minefield of subtlety. So this comes down to a choice of "pick the theory that gives the answer you like" or "trust the experts" (e.g. argument by authority).

That leaves us with the most popular option: compare the claim against some consensus (and it happens to be correct). This is generally easy, and works great when there _is_ a consensus, which leads us to overestimate its reliability. And thus we waste years exploring amyloid beta plaques, looking for dark matter, teaching whole-word reading, and so on.

It would be great if we had an easy way to tell who's lying, but in fact what we've got is a lot of ways to tell who we agree with and who we don't, and we don't always agree with each other on that.

OP here! Thanks for calling out this important point. As I fact-checked each claim, I was surprised at how many of the checks were "does the paper he's citing say what he says it does?" You can see them here: https://fact-check.brady.fyi/documents/3f744445-0703-4baf-89...

Yeah. And that's really important; If someone makes a correct claim by accident, say they misread a paper that incorrectly claims X as correctly claiming not-X, we shouldn't consider it evidence that they are trustworthy or honest, just lucky.

But then you have cases where someone correctly cites a source that they know to be incorrect (or at least plausibly should know). This is commonly done when flawed studies are funded specifically so they can be cited. This is arguably even more egregious lying, yet would pass a consistency based "fact check".

Likewise, the factual claim ("eight out of ten doctors surveyed recommend smoking brand-x") can be true while the implication is false.

In short, I'm not claiming such checks can't catch liars (they can), just that passing such checks doesn't mean they were telling the truth or what they said or implied was correct.