When did estimates turn into deadlines?

There is a spectrum of complexity: simple, complicated, complex. These can be framed in terms of ergodicity and you can search for Barry O'Reilly residuality theory if you want to go down this rabbit hole.

In a simple system we can easily predict the future states based on knowledge of past states. In a complicated system, we can also predict future states based on past, but it requires expert knowledge, though it's still fundamentally able to be understood (e.g. SpaceX rockets). These are both ergodic systems. Complex systems are non-ergodic.

Construction is a complicated system that exists within a complicated environment.

Software is a complicated system that exists within a complex environment.

Complex environments can be wrangled along three dimensions: constraining the environment until we can treat it as "only complicated", evolutionary survivorship via random stress and remediation, and avoiding commoditization.

Construction benefits from all three. Environments are constrained to enable the things we build to operate (cars work mostly on paved roads). There is a history of evolutionary survivorship spanning millennia. And construction is less easily commoditized (people typically do the same thing they did yesterday, and repeat for decades). All of this contributes to the ability to a better ballpark estimate.

Software primarily only attempts to constrain the computing environment in which the software runs: If you build your app to run in a container that can tolerate getting yanked and reincarnated elsewhere, you're golden, i.e. cloud computing is an example of constraining a complex environment until it can be treated as "only complicated". But the business environment remains complex and largely unconstrained. We attempt to constrain it via "give us your requirements", but that's more of an anchoring or negotiating technique than actually addressing the complex business environment. Software doesn't have millennia of evolutionary battle testing. And software is more easily commoditized, meaning if you do the same thing you did yesterday on repeat, that turns into a library or an app, so fundamentally you're always being pushed into novel territory, which therefore is less battle tested by evolutionary survivorship. All of this contributes to less clear estimates in software.

The issue isn't around tasks that are predictable in nature and therefore easy to estimate with a small margin of error, it's around complexity in software, unforeseen things, bugs, etc, which can compound for larger long term projects.

If engineers give estimates close to what it would be if everything goes right, then they risk overpromising and underdelivering if something goes wrong (hofstader's law). They might've just wanted to do the right thing by saving the company money and time, but in the end they footgunned themselves.

Or engineers intentionally over-estimate in order to manage the complexity, but then you end up with a lot of padding and parkinson's law. Because as soon as the engineer starts underpromising and overdelivering consistently, management will pressure them to lower their estimates because they have a track record of doing that, so instead they're incentivized to pad and then fill up the entire time they estimated even if it took less time.

Sprints were probably invented in order to deal with some of these issues, so that people just work with a bunch of smaller tickets that are much easier to estimate, with the more complex long term estimates going to management, which are incentivized to get it right because they're shareholders. That often leads to micromanagement and burnout, and it doesn't fix the padding/overestimation issue either, it might even amplify it in a lot of cases.

People here mention giving ranges or probability distributions, and have also equally mentioned that they don't work because management wants a single number, or management just assumes the best-case or middle-case of the range as the actual estimate, and then they still get in trouble for giving ranges and it didn't solve anything. It also doesn't solve the problem of unanticipated setbacks, the whole you don't know what you don't know thing, which can only really be solved culturally in some way.

While there are certainly bad managers that want to squeeze their workers, a lot of the time management is probably also pressured to give estimates and that's why they want and need that accuracy, because they're pressured by investors and clients that want to know how much time and money something will cost.

Overall the entire problem is a system cultural issue around managing complexity.