What factors explain the nature of software?

In civil engineering there are a lot of small creeks to cross that you'll pretty much use a drag-n-drop bridge solution for. When you make an estimate of how long it will take to build, it's most likely going to be correct.

It's when you come up to the wide river that you pull out the senior engineers and do a massive amount of homework before you even dig up the first shovel full of dirt. And even then your best laid plans are likely to go into cost and time overruns when one critical part gets delayed due to issues totally out of your hands.

I commonly find software far worse in pre-planning. People commonly bring large amounts of data together, then realize they have a computer science problem and are left scrambling on what to do, or cost surprises on how much computation is needed to process data at that scale/speed/latency.

Absolutely, but the author's explanation of why is both incomplete and unnecessary convoluted.

People pursue that which they can reason about, such as bike shedding[1]. Other answers are discarded without consideration. This is called bias. Perhaps the common bias failure that people employ about software are unfounded assertions. Developers make unfounded assertions typically in the quest for least effort without consideration of total cost of ownership[2]. Software managers frequently make the same errors for the purpose of reducing accounting expenses as opposed to personal time.

Senior executives and military leaders solve for this by surrounding themselves with advisors. Advisors occupy a managerial position of domain specific knowledge that eagerly seek to both keep their boss informed and simultaneously disqualify their boss's bad decisions. This works to eliminate bias so that the boss's job is then limited to balancing competing guidance from their advisors against their own managerial experience in consideration of attaining the stated goal.

[1] https://en.wikipedia.org/wiki/Law_of_triviality

[2] https://www.investopedia.com/terms/t/totalcostofownership.as...

In any high-level systems architecture, whether software or a chemical plant, you have a collection of notionally discrete and independent components that interface with each other. The properties of these components may be constrained in various ways e.g. the alloy type that can be used in a reactor vessel or the RAM available for a data structure. In a chemical system, that reactor vessel is modeled as a set of complex differential equations that govern its properties given many design inputs. In software we have much less sophisticated models of the architectural components; while there is a high-level concept of space- and time-complexity, those models are usually not coupled to a model of the hardware it is running on or temporal dynamics such that you can reliably predict absolute performance across multiple dimensions that are within a few percentage points of ground truth. With the current state of software engineering tools, there hasn’t been much immediate motivation to make the software component models much better than they currently are.

These models are the axioms of your system. In other engineering disciplines, there are sophisticated solvers that will take a large number of these components connected together arbitrarily with their many inputs and solve for the system with predictable and nearly optimal properties every time when you actually implement that design. You don’t have to guess as to the consequences and side-effects of changing requirements, scope, or implementation constraints, you can just re-run the solvers. The specification for what you want it to optimize for is pretty simple — cost, speed of building, throughput, etc. In software, we do almost none of this in a structured, disciplined way, not even manually. We know little about the properties of the design until after we’ve invested inordinate amounts of time building it first.

Ironically, it is much easier to build detailed and accurate component models in software than chemical engineering. We’ve somehow never gotten around to building systems design solvers in the same way they exist in other physical engineering disciplines. The behavior of the system is not the behavior of its isolated components when decoupled from the hardware environment, but in software we design most things as if this was the case.

Despite coming from a chemical engineering background, I didn’t fully recognize the existence of this gap until I started working in HPC. Now I see it everywhere in software. When I worked in HPC, I started hacking together crude models and solvers to address difficult optimization problems that took an excruciating amount of time to iterate on, a primitive version of what you might do as a chemical engineer, and it was a game changer. But it was an enormous amount of effort because there is no tooling to do this in software, even if it occurred to you that it might be a good idea for the same reason it is a good idea in every other engineering discipline.

I think we got away with it for a long time because early software systems were legitimately quite simple. We could live with it. Software is now vastly more complex. While our tooling has evolved to address problems of abstractions, it has not evolved to address the problem of systems behavior and modeling. The ability to see the effects of system design changes without actually implementing them is powerful. Other engineering disciplines recognized this need quite early; one of the earliest applications of computing was solvers for physical engineering systems problems.