Be good-argument-driven, not data-driven

Professors get this wrong all the time, despite being some of the smartest people we have around, despite decades of experience and education, despite a career and reputation on the line, and despite a system of peer review to catch mistakes before they get published.

Designing experiments is really difficult.

Interpreting experiments is difficult and unintuitive.

Statistics is difficult. You can't just look at whether the number went up. You need to have a deep understanding of significance, power and effect size, you should probably be doing ANOVA or some such.

You can create situations where you have a lot of data but can’t reach conclusions, because you lack a narrative and explanatory model which “makes sense” of that data; inversely, you can convincingly argue complete nonsense that’s obviously contrary to facts.

Deep understanding requires a model/narrative which fits the collection of data we have, and which allows us to reason about and predict the outcome of new situations.

As Jeff Bezos put it:

> Good inventors and designers deeply understand their customer. They spend tremendous energy developing that intuition. They study and understand many anecdotes rather than only the averages you’ll find on surveys. They live with the design.

> I’m not against beta testing or surveys. But you, the product or service owner, must understand the customer, have a vision, and love the offering. Then, beta testing and research can help you find your blind spots. A remarkable customer experience starts with heart, intuition, curiosity, play, guts, taste. You won’t find any of it in a survey.

https://www.aboutamazon.com/news/company-news/2016-letter-to...

For discussion's sake, let's go along with excluding data/metrics/science in pushing for arguments. In this framework, what exactly is a "good" argument based on? Gut feel? Opinion?

There was a famous quote by Jim Barksdale, the former CEO of Netscape: "If we have data, let’s look at the data. If all we have are opinions, let’s go with mine."

(So the tie-breaker in competing arguments in that case was "hierarchy-of-arguer-driven".)

So Jane and Bob disagree on the next action to take. Jane thinks her argument is a "good argument" but has no data. But Bob thinks he has a "good argument" but no data.

How does this thread's blog post help resolve the above scenario? (Blog's answer: you're driven by the one that has the good argument.) ... which is circular.

For disruptive innovation however, there needs to be an “argument” or opinion to help drive that data based on the industry trends. Companies then take a risk of delivering something new and good enough to the market. Also known as disruptive innovation.

This has shifted the idea of being data-driven to being one of “data-inspired”.

Anyone can make the same dataset fall into their favor. That’s the problem with being purely data-driven. Another way to think of it in the US especially is that our two party system makes wildly different conclusions from the same data. What’s preventing businesses from doing the same?

I highly recommend Conjectures if you can find a copy. It's a short read and interesting.