The other half of "Artists Ship"

That's the point I was trying to make. Something like "always commit and run the tests before going to production" makes a lot of sense, and should be mandated. Even something like "three signed copies of an authorization form" makes sense for some projects if, for example, the cost of damages (say, lost data, downtime or annoyed paying customers) is big enough.

Measuring this, though, is the hard part. It's like trying to do the math behind "no silver bullet" and estimate communication costs (too many unknowns and conditional dependencies).

Sure, yes, if e.g. the customers' health or safety was at risk. But I don't think the average bug in a web app would damage a company's reputation significantly. GMail occasionally shows me a message saying "Oops, an error occurred." It doesn't make me think any less of Google.

Imagine if, say, an Amazon customer service rep actually yelled, screamed, and abused a customer calling about his order.

A single individual - one who is not even in a high position. But you bet your buttons that it'll be all over the internet within ours.

Sure, it probably wouldn't even cause a blip to the stock price, but it's a huge amount of harm for a single person to cause.

I once worked for a company where anybody can push changes to prod without much feedback whatsoever. Certain more senior guys watched the commit logs for any obvious shenanigans or idiocy, but generally changes were within customers' hands within a day or two (and this was a desktop app!). They suffered some horrifying stability problem for a while that necessitated some new policies. It was certainly liberating to work in that environment, but I can see why many places have safeties in place.

Some things that PG didn't make explicit: web applications can have an extraordinarily tight feedback loop between customer and coder. Multiple revisions per day are easy, especially if any single user's data is rather low-value. Many a startup of this kind has zoomed past its corporate competitors simply by iterating faster. If your product is embedded firmware for a home security device, this strategy is just not available.

But here's another twist: when you start working for a giant company that provides myriad services under the same login, suddenly everyone has to move as slowly as the slowest part of the business. It doesn't matter if you are just doing Yippee! Backgammon, because who knows -- maybe you could somehow accidentally expose the data at Yippee! Payment Solutions.

This is a completely rational consequence of being a giant company and delivering multiple web services under the same login. It is possible to have slightly different authentication policies for each service. But I wonder if OpenID providers have thought about this enough.

I used an infrastructure site as an example because the value proposition is easy to understand when you use a site that has a clear and simple monetization strategy. Video sharing sites are arguably an even worse example than S3, because the value of uptime is so hard to perceive or compute. It's likely that even Twitter doesn't understand the true value of a customer-hour of Twitter uptime, because the site isn't monetized and so much of the value is concentrated in the brand. Measuring that is like voodoo, only less empirical. ;)

If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there.

No, but there is a big difference for you! If a user is worth a dollar per year, the five-user site is worth five bucks per year, but the million-user site is worth a million bucks. If each patch to your code causes 0.1% of users to abandon your product (a number which depends on the odds that a patch will cause a rollback, and on the odds that a rollback will annoy a user enough to make them leave), patching a 5-user site costs you half a cent per year on average (most likely it has no perceptable cost, since odds are no users will leave) but each patch to a million-user site costs you $1000 per year in revenue. And that's just the linear cost. There are nonlinear consequences: one or zero annoyed users is nothing to worry about -- unless that user is Michael Arrington -- but a clique of 1000 annoyed users is potentially a movement: a critical mass of people who will all start complaining about your company on Twitter on the same day, potentially costing you your next 10,000 or 100,000 or 1 million users while simultaneously empowering your competitors, who may begin building the site that will take you down by poaching those dissatisfied users.

This is just the flip side of scalability. As a programmer you enjoy mighty economies of scale: Running a site with a million users is more expensive than running a single-user site, but it is much less than a million times as expensive. But this leverage also applies to your mistakes: a mistake that costs you a dollar when your site is small might cost you $1,000,000 when your site is big. And it's the same mistake! Typos are just as easy to make on big sites as on small ones.

Obviously, this doesn't mean that you shouldn't ever change the site. Presumably each and every one of your patches is valuable, and will bring in revenue to pay for its own insurance premiums. Right? :) But you do need to think about that calculation, because you do occasionally make mistakes. As your userbase grows, you may wish to test each patch on a subset of users to be sure they will really like it, and that the additional revenue is really going to be there. You may wish to institute tests and internal audits that lower the risk of rollbacks, or failover mechanisms to lower the cost of rollbacks. And before long, lo, you will be that which you deplore: A company with a bunch of annoying internal controls! But at least you'll have revenue to console yourself with.