The McNamara fallacy: Measurement is not understanding

This seems untrue.

There are also genuine uncertainties about the future: things that neither side knows but will impact the outcome. Because of that, there is always a range of outcomes for a conflict, and so both sides would still have an incentive to carry out the war.

Consider, for instance, if there are two countries (A and B) looking at conflict with perfect information. Given their relative strengths, the war will cost both $1 billion, but A will win and reap $1 billion + $1 in plunder. In your model, B will always surrender and give A $1, but the reality is there is a big spread about both the costs and the gains. Lots depends on how well each side fights and responds, even with perfect knowledge of the other.

So...you know, if it's the US versus...the Philippines[1], then sure, the uncertain range of outcomes is small enough that rational thing for the Philippines to do is surrender. But for countries that are even reasonably well-matched, they would want to fight with the goal of making the conflict unattractive for the other side. Probably the best historical example of this is the USSR in Afghanistan.

[1] I am sure the US would never invade and occupy them just because the US found it convenient ;)

Quick story: I was the CFO for a company that sold to a private equity group (PEG). I took over as the CEO as the founders retired, leaving me to deal with the PEG. It quickly became apparent that the PEG managers looked at everything through the lens of an Excel spreadsheet. These guys were brilliant attorneys and analysts but lacked experience building businesses and managing teams. Ultimately, they couldn’t add much value in terms of operations or strategy, but they were great at financial modeling/quantitative analysis and forcing us to justify expenses. That may sound good at first—eliminating wasteful spending—but it ultimately led to the gradual erosion of the company culture and employee loyalty. It’s easy to cut benefits and pay given that many workers lack the leverage to do anything about it, while it’s much harder to reduce hard costs like materials and equipment. That meant employees just kept getting squeezed, and it was surprisingly difficult to quantify the impact that terminating an employee or cutting benefits would have on morale/culture/performance.

The moral of the story is that people with analyst mindsets play an essential role in our economy, but sometimes giving those people power over large organizations can have disastrous consequences. There truly is a disconnect between measurement and understanding.

> What McNamara didn’t keep track of was the narrative of the war, the meaning that it had both within the military forces of each side, but also in the civilian populations of the nations involved.

I'm probably not following and not saying it's easy but aren't there some metrics you could track that with? How was it determined/measured later that the meaning was important? Couldn't you at least do polls in your own country?

There are a bunch of factors and implications from the piece, which I had hoped they'd go into. But some of the more obvious implications which are often more organizational than they have to do with the data per se:

- the data you are collecting may not include factors that are consequential to outcomes. Treating these unmeasured factors as inconsequential is hazardous. The piece specifically mentions this effect. This effect is not a surprise but yet many organizations fall into this trap so it seems to be worth mentioning.

- the difficulty of measuring some factors will result in their exclusion from the dataset. Some things are intrinsically hard to measure, and many organizations will as a result refuse to measure them, and make decisions without them. There needs to be a conscious and active process organizationally to resist this and find effective ways of measuring them.

- the difficulty of measuring some factors will result in easier but less useful proxies being used in their place. Same as above just with a somewhat different outcome. Organizations are often blind to this happening as they come to believe the proxy is as good as the real measure (or sometimes even that the proxy is the measure). A good industry example of this is clickthrough rates being treated synonymously with audience interest or content quality.

And a point I wish the piece made but did not:

- measuring something does not grant automatic understanding of the phenomenon and gives you no predictive power. It's one thing to quantifiably know the blue button gets more clicks than the green button, but that grants you no insight into why. Many tech companies fall into this trap - where despite investing heavily in experimentation their modeling of the product space doesn't improve over time, since they fail to take the step to convert observation to generalizable hypotheses that improve their model for the product and market.

This last effect IMO is huge in our industry, and is why the same low-level experiments are being re-done over and over again. This creates more product churn and reduces your product velocity - since the lack of proven product models means you're mostly flying blind and using ex-post-facto experimentation on live users to figure out what to do.

Experimenting on button colors is all well and good, but the end result of that data should be a color theory that explains what colors to use when, not forever A/B testing every single button color for the rest of time.

But that’s the point - it’s not just how many, but what they are willing and able to do, their positional and operational strengths and weaknesses, and so on. Condensing all that into a number is impossible.