Perhaps I overemphasized it, but a personal experience on that front was key to realizing that the lesswrong community was in aggregate a bunch of bullshit sophistic larpers.
In short, some real world system had me asking a simply poised probabilities question. I eventually solved it. I learned two things as a result, one (which I kinda knew, but didn't 'know' before) is that the formal answer to even very simple question can be extremely complicated (e.g. asking for the inverse of a one line formal turning into a half page of extremely dense math), and two that many prominent members of the lesswrong community were completely clueless about the practice of the tools they advocate, not even knowing the most basic search keywords or realizing that there was little hope of most of their fans ever applying these tools to all but the absolute simplest questions.
> You can absolutely make meaningful predictions about the world despite uncertainties. A good model can tell you that a hurricane might
Thanks for the example though-- reasoning about hurricanes is the result of decades of research by thousands of people, the inputs involve data from thousands of weather stations including floating buoys, multiple satellites, and aircraft that fly through the storms to get data. The calculations include numerous empirically derived constants that provide averages for unmeasureable quantities for inputs that the models need plus adhoc corrections to fit model outputs to previously observed behavior.
And the results, while extremely useful, are vague and not particularly precise-- there are many questions they can't answer.
While it is a calculation, it is very much an example of empiracy being primary over reason.
And if someone is thinking that our success with hurricane modeling tells them anything about their ability to 'reason things out' from their own life, without decades of experience, data collection, satellite monitoring, teams of PHD, then they're just mistaken. It's just not comparable.
Reasoning things out, with or without the aid of data, can absolutely be of use. But that utility is bounded by the quality of our data, our understanding of the world, errors in our reasoning process, etc. And people do engage in that level of reasoning all the time. But it's not more primary than it is because of the significant and serious limitations.
I suspect that the effort require to calculate things out also comes with a big risk of overconfidence. Like, stick your thumb in the air, make some rough cash flow calculations, etc. That's a good call and probably captures the vast majority of predictive power for some new business. But if instead you make some complicated multi-agent computational model of the business it might only have a little be more predictive power but a lot more risk of following it off a cliff when experience is suggesting the predictions were wrong.
> people do things in dumb inefficient ways all the time
Or, even more often, they're optimizing for a goal different than yours, one that might not even be legible to you!
> just as that model predicted it would be, because I did the math and my competitors in a crowded space did not.
or so you think! Often organizations fail to do "obvious" things because there are considerations that just aren't visible or relevant to outsiders, rather than any failure of reasoning.
For example, I've been part of an org that could have pivoted to a different product and made more money... but doing so would have meant laying off a bunch of people that everyone really liked working with. The extra money wasn't worth it. Whomever eventually scooped up that business might have thought they were smart for seeing it where we didn't, but if so they'd be wrong about why we didn't do it. We saw the opportunity and just had different objectives.
I wouldn't for a moment argue that collections of people don't do stupid things, they do-- but there is a lot less stupid than you might assume on first analysis.
> it's just that that usually happens in areas where you have singularly deep expertise, not where you were just a Really Smart Dude and thought super hard about philosophy
We agree completely there-- but it's really about the data and expertise. Sure, you have to do the thinking to connect the dots, and then have the courage and conviction (or hunger) to execute on it. You may need all three of data, expertise, and fancy calculations. But the third is sometimes optional and the former two are almost never optional and usually can only be replaced by luck, not 'reasoning'.