How I estimate work

Incorrect analogy. Bridge construction is a clearly algorithmic process. All bridges resemble each other, and from an engineering perspective, designing one is not rocket science. Construction itself is a set of well-studied steps that can be easily calculated. If I were to write my operating system 100 times, I could give an estimate accurate to within 10%, but every task I’ve ever done in life is unique, and I have nothing to compare it to except intuitive judgments. Returning to bridges: there is 1% of projects that are unique, and their design can take decades, while construction might not even begin

Not a single mention of planning poker and story points?

They're not perfect (nothing is), but they're actually pretty good. Every task has to be completable within a sprint. If it's not, you break it down until you have a part that you expect is. Everyone has to unanimously agree on how many points a particular story (task) is worth. The process of coming to unanimous agreement is the difficult part, and where the real value lies. Someone says "3 points", and someone points out they haven't thought about how it will require X, Y, and Z. Someone else says "40 points" and they're asked to explain and it turns out they misunderstood the feature entirely. After somewhere from 2 to 20 minutes, everyone has tried to think about all the gotchas and all the ways it might be done more easily, and you come up with an estimate. History tells you how many points you usually deliver per sprint, and after a few months the team usually gets pretty accurate to within +/- 10% or so, since underestimation on one story gets balanced by overestimation on another.

It's not magic. It prevents you from estimating things longer than a sprint, because it assumes that's impossible. But it does ensure that you're constantly delivering value at a steady pace, and that you revisit the cost/benefit tradeoff of each new piece of work at every sprint, so you're not blindsided by everything being 10x or 20x slower than expected after 3 or 6 months.

I am sure you make this work on good faith at a small size, but how is it doable at large without insider trading, manipulation, and perverse incentives? People doing the work will be more informed and in a position to affect the outcome.

> What often baffles me with engineers and especially engineering managers is that they don't derive the estimates from metrics of prior projects, especially for long running teams (as opposed to project teams).

The PMI project management methodology even pre-scribes this (after the project is over, to reflect, and then to gather collected learnings and objective stats so as to help better estimation of future projects), but the problem is many engineers see project management a administrative burden rather than as a tool, and they have a tinkerer's mind-set rather than a scientist's mind-set.

Good project management practice is also not to use point estimates but intervals: something is going to take "betweek [i;j] person days". PERT-estimates are even three-point estimates of work (best case, expected case, worst case), so you model the uncertainty explicitly rather than by some alchemist formula ("double and add 20%").

Incidentally, I found out empirically that the best technical people tend to be the worst at estimating their own time needed to complete a task. That's because the best are often also a bit over-confident, perhaps. Letting each team member estimate a task and adding a 20% correction for unforseen issues has worked well to make my projects be delivered on time.

Tip: Push back if management tells you how long you have; either explain to them what you can give to them in the time they dictate or reject their task order and say you will come back when you have calculated how long the project takes, which can only be rationally determined once the scope is clear(er).

Check out/google: PMI PMP methodology and also PMBOK (project management body of knowledge) > "Organizational Process Assets" (OPAs) > "Lessons Learned Repository"

Usually less than 2 days, with a 90% confidence.