Dear Agile, I’m Tired of Pretending (2018)

In addition to the difficulty in making a meaningful estimate, I'm frustrated by how often management really wants to hear a particular answer and won't settle for anything but that answer. It can go like this:

1. Developers size development of X and says it will take roughly 12 months.

2. Management says the rest of the organization is going into system test in 9 months and needs X finished by then. Couldn't there be some way of speeding up the development of X without impacting the rest of the organization?

3. X's developers say no, not really.

Now management tries one or more of the following:

-- find someone else to size the work that X's developers will be expected to do

-- hire some outside "expert" to size X

-- hire contract programmers that claim to be experts in X that will join the X team

-- suggest reduced functionality for X

-- propose some modified "streamlined" component testing for X

-- promise to provide X's required dependencies ahead of their scheduled completion dates so X's development won't be gated by these requirements

Now jump back to step 1. Repeat until X's team relents and says that maybe, just perhaps they might be able to finish X in 9 months if everything goes perfectly even though the company has never done X before

This is now passed up the management chain and it becomes part of the official overall system's schedule that planning, marketing, sales, finance, etc. is depending on.

Iterating over all of these estimates is a big waste of time. Management has already decided that they want X in the system and will keep pushing until an estimate comes back that they accept.

If X's team is lucky, some other team will fail to make their estimated completion date and X will end up with the 12 months it might take to finish X. Every team in the organization is hoping that it will be a different team that first has to admit that they will not make their date. This causes a kind of game of chicken where every manager is unwilling to be the first to admit that their team is going to miss it date. Consequently, every date in the organization cannot be relied upon. The distributed file system needs the TCP stack to be finished on time, but the TCP stack needs the ethernet device drivers to be finished on time and no one will admit slippage of any dates first.

I've become convinced over the years that upper management doesn't really care about 'estimates'. What they care about 'commitments'.

Every team I've been on has had upper management drive team leaders to get 'commitments'. They want some form of emotional investment in the work. That way, when scope has arbitrarily changed or a massive problem has lead to run over, they can stare at you like a whipped puppy-dog and say "I know it's 10PM and you want to get home to your wife, but you promised us, you committed, to getting this done on time..."

Pure emotional exploitation. The ideals of agile are laudable. I was the excited bannerman of my first agile experience and I read the manifesto with a lot of hope. However high-minded agile is, the language is too easily co-oped into the worst sins of management. In many places it exists solely to blackmail free overtime out of engineers, get away with micro-management, and ride engineers with false deadlines under a guise of hipness and modernity.

Estimates are just there for a false sense of stakeholding, or at best a ballpark cap so you don't work yourself into suicide. They can't be made accurately even if we could estimate properly, because all estimates are constantly second guessed by people who are making judgmental jokes about sandbagging, or asking "Are you really sure it'll take that long?" in voices that practically scream "chilling effect". More than a few possibly realistic estimates are torpedo'd by managers who are looking at their roadmap and tsking about how they don't line up.

I would even say accurate estimates are actively undermined. I've worked at companies which have positive work images abroad, and even there any team which was on time found either its sprint budget slashed, or scope increased until it was forced behind. The underlings won't do over-time if they don't feel the squeeze.

Story points, velocity, burndown, and any other metric you can think of that Agile courts are absolutely useless, because this industry refuses to come to terms with Goodhart's Law. I just watched a 10 million dollar project go up in flames. Their sprint graphs were great, and a constant buzz was going about the company about how these teams were setting the standard, meeting their metrics and goals consistently. Only recently did the CEO get down to business and figure out that all the metrics were bogus, and that for the past year all of the PMs and teams had been gaming the system to hide the fact that they were way behind schedule and over budget. The project was quietly scrapped and nothing changed. The shell games continue to actively sabotage data. I only know of the fallout because of who I sit next to.

Most of the benefits of agile seem to fail basic contact with humans unless they are backed by outstanding and visionary leadership. Most companies do not have that, yet they still find value in switching to Agile. Why? Maybe because even in total collapse it provides an unparalleled system for squeezing out more overtime, in my cynical opinion anyways.

I don't really what else to offer in place of Agile. I'm not that intelligent, but I don't really feel the industry can come up with a good development model of software development culture until it stops what it's doing and starts acknowledging problems arising from basic psychology, politics, and data gathering.

Assuming you are not being sarcastic, quote from another post;

>the old trope of using something meaningless but easy-to-measure as a proxy for something important but hard-to-measure

The Why rather than the What. Maybe they don't even need a button.

It just sounded like teasing because it's a famously hard problem.

But having read your proposal, I actually totally believe that that could result in accurate time estimates. To summarize, remove the unknowns by planning things out beforehand, then count how many hundreds of small pieces you'll need! That sounds plausible as long as what you're doing is straightforward. (E.g., no crazy database optimizations needed to make things fast enough, no crazy ML techniques needed to make things accurate enough, no crazy algorithms needed to solve NP-hard problems, etc.)