‘Strong Opinions, Weakly Held’ Doesn't Work That Well

I agree, but that doesn't make the "old data" problem go away.

It's worth noticing that in the first part (Paul Saffo forecasting), the problem is all in one head. Saffo recognizes an unintuitive methodology: strong opinions weakly held. He's only really dealing with himself, his intuitions, tactics that work for him.

The second part gets social:

"you’ve decided, along with your boss, to build a particular type of product for a particular subsection of the self-service checkout market."

This is a good example, imo. These decisions are made with imperfect information, and they incorporate a lot of interlocking opinions. Ex ante can only get you so far. You need to make a decision and start rowing to progress, change paths when necessary.

Multiple people are involved. Credibility is on the line, among many subtle subroutines of group psychology that we aren't even aware of. The solution from Tetlock, express your opinion as a probability, is a solution to the problem in its group form. When people are in meetings making big decisions, a lot more than the big decisions is going on. Human power dynamics are taking place. Being right can play a very small role in these.

I think the author has a blindspot. "This is not how the human brain works" is one kind of challenge. "This is not how human groups work" is another. It isn't a derivative of the first.

Baseline political paradox - humans demand extraverted confidence from leaders more than they demand competence.

If you're confident you can make mistake after mistake and no one will care unless everything comes crashing down - and even then there's a fair chance you can make it someone else's fault.

If you're merely very competent, taking time to research good solutions looks like indecision, not leadership, even if the results are far better.

There is a sort of financial epistemology that a lot of people like because it tells them they are successful and therefore right. It also allows them to talk about testing hypotheses and get results in hard numbers. And it even works as far as it does, but I’m more and more skeptical that it explains everything.

Of course, "strong opinions weakly held" is a refactoring of the scientific approach to knowledge. Karl Popper taught that scientific hypotheses necessarily must be disprovable. Hypotheses that aren't disprovable (falsifiable) are useless in science.

Insisting on falsifiable hypotheses is a difficult personal discipline, not to mention collective discipline. "Nothing is true unless it might be false!" Wait, what? How can a company make decisions based on that sort of epistemology? How can a government set policy? It's much easier to make decisions when we have the illusion we're sure about the facts.

Maybe the only sensible path is to hold all our opinions, personal and group, lightly.

I think this might be due to a difference in optics. What you see as 'asking for a second or third opinion' might be seen by others as lack of independence, leadership and ownership.

Mostly everywhere senior developers are expected to self-manage and make difficult decisions on their own, unprompted, and just present working solutions to the business. It's okay to take time on this - solving it on your own, consulting literature, peers, etc. But looping in management into technical decision making might make you look like you can't make decisions on your own, or need micromanagement. Management, business and product care about things being done - not technical approaches or having to choose between multiple solutions.