Why Twitter didn’t go down: From a real Twitter SRE

First of all, how does it persuade you of that? The article touches a really small (though incredibly important for up-time) subject.

Secondly, in any large company, the majority is 'bloat'. It's security engineers, code reviews, data architecture, HR, internal audit teams, content moderators, ccrum masters and I can keep going. In a start-up many of these roles can be ignored, becaus growth > stability. In a large organization, part of the bloat helps insure a certain amount of stability that's necessary to keep an organization alive.

If a product is mature enough, like Twitter seems to be, removing engineers won't instantly crash the product. It'll happen slowly. Bugs will creep in, because less time is spent on review and over all architecture. Security issues will creep in because of about the same issues and less oversight. Then, once this causes enough issues for the product to actually crash, the right people to fix it quickly might not be there anymore. That's when fixing the issues suddenly takes a lot more time.

If the current state of affairs at Twitter keeps up, it'll probably be a slow descent into chaos. Especially with Elon pushing for new features to be implemented quickly, inevitably by people who cannot fully understand the implications of said features, because 80% of knowledge is missing.

In my experience as a data engineer, unusual states are one of the leading causes of issues, at least after something is built for the first time. You can spend half a year running into weird corner cases like "this thing we assumed had to always be a number apparently can arbitrarily get filled in with a string, now everything is broken."

Also, conditions changing causing code changes is the norm, not the exception, definitely in the beginning but also often later. Most services aren't written and done - they evolve as user needs evolve and the world evolves.

From today's Casey Newton's newsletter:

In early December, a number of Twitter’s security certificates are set to expire — particularly those that power various back-end functions of the site. (“Certs,” as they are usually called, serve to reassure users that the website they are visiting is authentic. Without proper certs, a modern web browser will refuse to establish the connection or warn users not to visit the site). Failure to renew these certs could make Twitter inaccessible for most users for some period of time.

We’re told by some members of Twitter’s engineering team that the people responsible for renewing these certs have largely resigned — raising concerns that Twitter’s site could go down without the people on hand to bring it back. Others have told us that the renewal process is largely automated, and such a failure is highly unlikely. But the issue keeps coming up in conversations we have with current and former employees.

Aren't these changes inevitable, though? There is no such thing as bug free code.

Another thing that forces consistent code changes is compliance reasons- any time a 0-day is discovered or some library we're using comes out with a critical fix, we would have to go update things that hadn't been touched sometimes in years.

At my last job, I spent a significant amount of time just re-learning how to update and deploy services that somebody who left the company years ago wrote, usually with little-to-no documentation. And yes, things broke when we would deploy the service anew, but we were beholden to government agencies to make the changes or else lose our certifications to do business with them.

Eventually, Twitter will have to push code changes, if only to patch security vulnerabilities. Just waiting for another Heartbleed to come around...