How big tech runs tech projects and the curious absence of Scrum

ESR wrote http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral... in 01997, but Jim McCarthy said "Release Early, Release Often" in his 01995 software-process book https://www.amazon.com/dp/1556158238 (which I haven't read) according to https://devops.stackexchange.com/a/80.

Releasing often isn't always a good idea, but it isn't necessary for every user to upgrade to every release. Here are some advantages of releasing every iteration (once or twice a month):

1. You can never have the software in an uncompilable or unshippable state for an entire iteration, much less three months at a time, so the only time you don't know how long a showstopper bug will take to fix is immediately after you find it, and you're never "integrating, don't know how long this will take," because frequent releases force continuous integration. You never slip a ship date, at least not by more than a week or so; you just cut scope. Slipping can be enormously painful and sometimes fatal to a business.

2. A hacker costs a business US$200k a year. A team of ten hackers costs US$38000 per week. If you release once a quarter, then, on average, at any given moment, you have six weeks of work that you've paid for that haven't reached users and are thus not yet producing any value; that's about a quarter of a million dollars of "inventory"—having a lot of inventory might sound great if you're a survivalist or an RPG player, but for a business, "inventory" is wasted money. If you can cut that to monthly, you only average two weeks of inventory, so you've saved US$170k, almost the cost of a person-year of work.

3. When you ship software to users, they react, which is an opportunity to learn from them about what they value and what they don't; this both gives you ideas for new features and gives you more information about the value of features you'd already thought of. This is true even if only some of your users install the new release. If you ship once a quarter, you get four of these round trips per year; if you ship once a month, you get twelve. So you can learn a lot more from your users. Evidently that wasn't enough to get Skype on track.

4. If you're a business, the business environment changes as you're developing software: for example, your competitors add new features, systems you interoperate with change their behavior in ways that break your software, and new libraries become available. To remain profitable and in particular to remain better than the competition, you must observe these changes, orient yourself with respect to them, decide what to do, and then act. In competitive environments, the competitor who can go through this "OODA" loop faster will almost invariably win. Shipping features that embody the right decisions for this month instead of the right decisions for three months ago can be a decisive advantage—though I think this is what WhatsApp was better at.

5. When your sales guy meets with a potential client on October 10 and finds out about a missing feature that's a showstopper for that client, and the October 28 release now has that feature, it gives you totally different chances of closing the deal than if the January 10 release has the feature. Even if they don't buy then, you've made a totally different impression, and you're more likely to make a sale later or get a lead on another potential client from them.

6. More generally, frequent releases make progress and quality legible to people outside the programming team, circumventing the "thermocline of truth" problem named in https://brucefwebster.com/2008/04/15/the-wetware-crisis-the-.... Having five features "80% done" after four months could mean you'll deliver them in another month, or that you'll deliver them in another 8 months. By contrast, the difference between delivering four of those features in four monthly releases, and delivering one of them, is a difference observable to users, management, sales, and customers. Sometime it's even helpful for keeping the people inside the development team from fooling themselves, despite being below the thermocline of truth.

I'm writing this based on the experience of utterly smashing two established competitors at AirWave, where upgrading one of our clients always involved a field engineer, often doing a site visit. We weren't writing open-source software, either.

Seriously, though, having your software enforce too much prescriptive workflow on you is dramatically worse than having it enforce too little. Historically it was common for every software team to write their own bug tracker for this reason.

· nightly builds or continuous integration;

· pair programming or code review;

· automated builds;

· comprehensive test suites or test-first programming;

· fixing bugs before adding features;

· source code version tracking software;

· keeping a list of bugs and planned features instead of not keeping one;

· prioritizing planned features instead of not prioritizing them;

· merciless refactoring to keep the design simple;

· DRY, as a criterion for what "simple" means;

· "YAGNI" (not writing code to provide functionality that I'm not implementing right now);

· a sustainable pace (no death marches);

· exploratory "spikes" to explore unknown features;

· taking regular breaks;

· informal usability tests with prototypes;

· standup meetings;

· frequent releases;

· quiet working conditions; and

· retrospectives.

I'd add, though these usually go without saying these days:

· high-level languages;

· interactive development environments, as opposed to the batch-mode approach where you submit a batch job and get back your program output an hour or a week later;

· enough testing hardware that you never have to wait for a testing machine to become available in order to get your work done;

· using available software libraries instead of not using them.

I'm pretty confident that each and every one of these helps me build better software faster, though a few of them may vary somewhat depending on circumstances—I know a guy who can concentrate to write code better in a noisy café than in a private office, frequent releases are at best minimally useful for pacemaker firmware, and you couldn't have written qmail using existing libraries.

Also, though, I'm guessing that the old codebases you're looking at with simple spreadsheets for bug lists practiced "working software over comprehensive documentation", "customer collaboration over contract negotiation", "responding to change over following a plan", and especially "individuals and interactions over processes and tools". And those are the core aspects of agile development (though not the Fake Agile we so often see).

Some of these are things earlier generations of programmers could only rarely do. If you're working at a company that values processes and tools over individuals and interactions, there's not that much you can do to fix that if you couldn't start your own company; https://news.ycombinator.com/item?id=28670326 suggests one reason this is so widespread. Similarly for fixing bugs before adding features, using an interactive computer, having adequate testing hardware, and having quiet working conditions. Source control systems, automated builds, and automated test suites provided substantial benefits to teams that adopted them, but many didn't. That doesn't mean they weren't valuable at the time, just that many people did without.

By contrast, I'm less convinced of the value of private offices, written spec documents, schedules, dedicated low-paid testers, scrum masters, planning poker, splitting and merging product backlog items, quantitative task estimation, and even coding interviews.