See also the book How to lie with statistics and similar (I think a follow up book was called How to lie with charts and graphs).
Be good-argument-driven, not data-driven
21–30 of 168 posts
Re: Be good-argument-driven, not data-driven
#22The hidden assumption here is that things go well if and only if (you think) you understand all the factors that influence your metrics, can do experiments and are prepared to use fancy statistics.
Which I reckon is a bit iffy. Special relativity was thought out well before any experiments to test it were feasible, and if understanding everything that influences your metric is a prerequisite then you can blame all failures on insufficient understanding without having any way of knowing when you have enough understanding.
Re: Be good-argument-driven, not data-driven
#23While I agree completely with the premise of this article, on the other hand I'm weighing the relatively robust findings by Meehl et al. They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment[1]. One would think this is cognitively dissonant enough, but it gets worse: This article, with the th…
I find it strange that these are presented in tension, when they’re complementary. You can create situations where you have a lot of data but can’t reach conclusions, because you lack a narrative and explanatory model which “makes sense” of that data; inversely, you can convincingly argue complete nonsense that’s obviously contrary to facts. Deep understanding requires a model/narrative which fits the collection of d…
I was about to write that in case of Bezos with Amazon, the customer was simpler and the answer was to just pour money into it until you substituted the market, but I realise now that that is not that simple. It seems simple because we have hindsight.
My main idea though is that it is very hard to foresee what the customer will want after you deliver the product. Not what the customers want now, because sometimes they don't understand it until they experience it, and that makes me think that there is a LOT of luck at play here and a good deal of continency in prototype product design. Experience alone could be overrated. Think Kodak, I don't think they didn't have experience in product design, that they didn't understand their customers. I think they only didn't risk their luck and didn't think about what their customers would want in the future. And that is always a gamble.
- Things are more nuanced and complex than I am putting it here, but bottom line is that I am trying to tap into survivors bias.
Re: Be good-argument-driven, not data-driven
#24Earlier quoted context omitted.
> Observation is always selective. It needs a chosen object, a definite task, an interest, a point of view, a problem. And its description presupposes a descriptive language ... which in its turn presupposes interests, points of view, and problems. Thanks, I'd never heard this quote before. He's pretty much describing pragmatism à la William James. I had no idea.
The pragmatists went a little bit too far in my opinion, though it has been a long time since I read any of them. Popper is describing observations, not reality. I highly recommend Conjectures if you can find a copy. It's a short read and interesting.
What do you mean that they went too far? James and Peirce were not describing "reality" (in this discussion anyway. [1][2]) but rather were instrumentalists and thus saw every theory as having a purpose. That's the whole point of the squirrel argument. It not just "depends on what you mean" (as per analytic and some medieval philosophy) but also depends on what you're trying to do (which in turn depends on what you want/like.) In any case, the similarity I was pointing out is just that theories have purposes and ignoring this is a blatant blunder.
1. James even endorsed religion and other make-believe if it was useful to your purposes.
2. Peirce: "Consider the practical effects of the objects of your conception. Then, your conception of those effects is the whole of your conception of the object."
Re: Be good-argument-driven, not data-driven
#25One of the big reasons why data driven approaches are so seductive is, it's very difficult in the moment to distinguish between a good argument and a well crafted rationalization.
The issue is that it doesn't fundamentally solve the problem. It's true that a good argument logically supported by data is better than a good argument that hasn't been checked against data. But the existence of data in the argument doesn't help you determine whether it's a good argument logically supported by data, or a well-crafted rationalization speciously supported by data.
Re: Be good-argument-driven, not data-driven
#26While I agree completely with the premise of this article, on the other hand I'm weighing the relatively robust findings by Meehl et al. They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment[1]. One would think this is cognitively dissonant enough, but it gets worse: This article, with the th…
Seems far-fetched to assume that this thesis applies to product development just the same?
The impact a data-driven mindset can have on the organization cannot be understated ('RIP intrinsic motivation' section). I've seen it first-hand, both data being used as cop-out for bad leadership, meaningless 'successes' used as trading cards for promotions, and design experts having a decade of experience overridden by shaky statistical analysis, or worse, non-inferiority tests.
Meanwhile, the shortcomings in the product that everyone knows are rarely addressed because they are 'difficult to test'.
Re: Be good-argument-driven, not data-driven
#27While I agree completely with the premise of this article, on the other hand I'm weighing the relatively robust findings by Meehl et al. They find, time and time again, in all sorts of fields, that extremely parsimonious models like equal-weighted linear regression of one or two predictors outperform expert judgment[1]. One would think this is cognitively dissonant enough, but it gets worse: This article, with the th…
I've come to heavily discount these types of studies. What makes an expert? What was the sample size of experts? What was the non-expert tool? Etc.
There is such a thing as having common sense based on thoughtful life experience. Checklists and regressions help, but human beings are very capable of deep expertise and to pretend otherwise is silly. I expect a musician to be able to identify a violin from a viola.
Re: Be good-argument-driven, not data-driven
#28"According to the data on business failures, you should have never started this business"
Re: Be good-argument-driven, not data-driven
#29> Are you prepared to do some very very fancy statistics?
I'd extend this with "... while understanding what you're doing?"
I've seen it so many times already, someone does some A/B-test and then presents a very fancy looking slide-deck with all kinds of crazy-looking math. But if you start to ask questions, it's all very obvious that they didn't really understood what they were doing and that very often it doesn't really matter to them in the first place; it's all about reaching a decision using some pseudo-scienty method that nobody dares to question because 'data' and 'science', without having to take responsibility.
Re: Be good-argument-driven, not data-driven
#30> A weak argument founded on poorly-interpreted data is not better than a well-reasoned argument founded on observation and theory. So a good argument is founded on...good data and good understanding of data? The article more seriously makes the mistake of begging the question: it presupposes the known classier of good and bad arguments and then goes on to say bad arguments with data is worse than good arguments. But…
> It presupposes the known classier of good and bad arguments and then goes on to say bad arguments with data is worse than good arguments.
It does indeed assume that there's a way to learn bad arguments from good; and so the focus should be on learning what are good argument and what are bad.
> ...What makes a good argument if not empirical data?
Consider the following conversation:
A: We've done some numbers, and we've determined that there's a correlation between the number of firemen at a fire and the total damage done by the fire; with the fires handled by a single crew of three firemen doing the least damage. So we should limit all fire responses to a single crew to minimize damage.
B: That doesn't make any sense -- of course we send more firemen to bigger fires, and bigger fires cause more destruction! If we take your advice, those big fires will cause even more damage!
A: Hey, my argument is backed by empirical data; yours is just theoretical!
Like, sure, it might be even better if B had empirical data to back him up; but even without that data, B should be winning the argument here. And the argument of the article is that many people espousing "data-driven" approaches end up being like A: Not scrutinizing the logic that they're using to analyze the data, and not acknowledging the limitations of what the data collected can say.