Innovation heroes are a sign of a dysfunctional organization

I experienced something like this for many years across several of the big five defense contractors and smaller SBIR contractors working for the US DoD and IC. As someone who 20 years ago had a slightly better understanding of how computers and networks actually work than an average developer and was comfortable at the command line in an era when juniors increasingly couldn't leave their IDE without becoming hopelessly lost, as DevOps, SRE, and platform engineering started to become things, my career drifted in that direction.

The problem being on teams like that was always the same. You're responsible for developing software products of some sort, but they're software that runs, tests, or delivers other software or even orchestrates the operations of an entire environment shared by many different applications. This inevitably means a whole lot of your work is the classic incident response/post-mortem of operations, plus some level of customer support given to other development teams because internal platform teams never get a separate support organization. To this day, the government has no idea how to handle this in light of how DFARS (administrative law governing acquisition) works.

Every labor hour a contractor charges has to be tied to a specific line of accounting which is itself tied to some unit of planned work tracked in an issue tracker or project management system of some sort. Most of the time, this is an Epic in Jira. This is logical insofar as you consider the intention of acquisition as a category of appropriations bill. A budget proposal with line items tied to measurable product features is presented to and approved by Congress. You have to demonstrate you're spending the money they gave you on what they approved you to spend it on, because per the Constitution, that's how power of the purse works. The executive branch can't just do whatever it wants.

But it falls apart as contractor labor increasingly replaces federal civil servants. When your job is anywhere from half to all running and maintaining an operational system, that isn't really acquisitions any more. It's operations and maintenance, which is an entirely separate appropriation. Soldiers manning a defensive perimeter have no idea when they're going to be attacked or how much work they'll be putting in. Software operations is effectively just the less lethal civilian equivalent of that. The government arguably even recognizes this in many ways. Typically, running and maintaining the production system is a separate contract from the development contract and it uses separate work tracking systems, and if someone spends 8 hours watching a screen while producing no quantifiable work outputs, so be it. They charge 8 hours because that's what the contract says they're supposed to do.

But as we're increasingly expected to be modern software organizations with things like CI/CD, test and staging environments, and you inevitably need to run at least some of your own development infrastructure, well now what? You have a lot of people whose job is the same. Be on constant lookout and respond as needed, but now it's part of a development contract and they need to have quantifiable work outputs that be tied to a budgetary line item with an associated product feature.

So we convolute nonsense out of thin air like cloning the same monthly "Support" Epic in Jira, month after month. "As a developer, I want my tooling to work so I can do my job and deliver value to the government." Plug in some SWAG number vaguely guessed at based on how much of this kind of work you ended up doing last year. Every 90 days, play planning poker with it. LARP a product team even though that isn't what you are.

Careful, though, because you may encounter someone who can sniff out these tricks.

I worked at a place where, like many companies, tracked the bug burndown as one of the measures of project risk. So, the closer you got to the release, the fewer bugs were expected to be open. Having many bugs open early in the project was fine and encouraged. Having few bugs open towards the end was seen as On-track, and having many bugs open towards the end was a red flag, where the program might want to either step in and help organize things or at least raise the risk with higher management. This is a pretty standard scenario you get at a lot of software companies.

Well, one of the teams realized that if they just kept their bug count always at zero, they'd never be bothered by those nasty, annoying project managers. So that's what they did. They were constantly in crisis, fixing problems and committing code, but their bug counts always looked great. A trivial bug tracker query revealed what they were doing: They'd just keep a side bug tracker in a doc rather than filing them in the real tracker, and then seconds before committing the code, they'd quick create a bug and use that as the required bug in the commit message.

So, the following week, these "ninja bugs" were added to the automated reports, and that behavior ended quickly.

Don’t forget incentive structures that award developers for “org wide impact”. Often the way to do that is to roll your own crap instead of use off the shelf stuff. This is how you wind up with developers creating their own key-value databases or billing systems… it sounds much better to write these from scratch.

A backhoe is orders of magnitude more complex than a shovel. A team struggling to dig a pit with shovels would probably benefit immensely from a backhoe. The idea that workers shouldn't get better tools until they can succeed without them is insane.

Counterpoint: most innovation sucks. Sure, innovation is necessary for improvement and adaptation to changing circumstances. Sure, sometimes innovative change is stifled to the point that organizations crumble under the stasis. But the larger and more stable the organization, the higher the bar should be between “I have a great idea that will make things better!” and its implementation. Most ideas of change do not take into account the complex, multiplicitous, overlapping, crossed-purpose systems that conspire to bring an organization to life. They account for only locally perceived changes, not weird, unpredictable knock-on effects that can arise from introducing an optimization. Adopting every ostensible innovation that comes along is inviting only chaos. Change should be hard. Not impossible, but hard.