Post-mortem of Christmas Eve Amazon ELB outage

As usual, their summaries leave me with more questions than answers.

A maintenance process whose purpose in life is to delete data from ELB backend database (if it were not the case, you'd see "maint process didn't work right") in such a weird way that it would cause such chaos? Why on earth would such a maint process exist in the first place? I can imagine some possibilities here of course but it feels to me there is more to it than what they've chosen to disclose.

Next. So they lose config data but data path for now not impacted. Fine, makes sense. But then backend, with only partial data, attempts to reconfig running LBs, doesn't fail completely (as in it was able to connect and do at least something but not all actions it was supposed to do) thus forcing otherwise good but impaired LBs into a completely bad state. Sunds suspicious to me.

And then the biggest question - why did they choose to attempt to restore entire backend database when only 6.8% of LBs were impacted?

I also have no idea how a CM process can protect against making a mistake - mistakes happen when somebody is at the controls with or without prior coordination.

All in all, their backend systems are so sophisticated and precisely engineered that any unforeseen/unexpected abnormality caused by manual intervention (be it inadvertent run of a maint script or fat fingered traffic re-routing from primary to backup network) inevitably lead to overwhelming reaction of their automation that makes the problem even worse and extremely hard to recover from.

Very tough position to be in - during these outages, they are essentially fighting the skynet that they themselves created and at their scale there is no way around it.

So hats off to those who've been working on this and good luck taming the beast.

Wouldn't it be a better idea to learn from a provider that isn't constantly screwing things up? Amazon's systems clearly aren't working reliably.

If Amazon can have an issue of this severity in ELB, one of their core core services, during peak traffic season for their #1 highest profile customer, every one of their services should be viewed with suspicion and you really need to have non-amazon backup systems in place in case they have an incident that affects you. Yes it is painful to leave the comfortable aws womb, but it's time to grow up and start either managing multiple infrastructure providers or build your own.

There are any number of ways a number could be considered "special". It could be prime, it could have no repeated digits, it could have sequential digits, it could be a friendly number, it could be a palindrome, it could be a perfect number, ... I could go on...

Eventually the numbers with nothing "special" about them begin to appear the minority: http://www2.stetson.edu/~efriedma/numbers.html

I think the measures they implement to prevent this failure are adequate. Changing access control patterns so that not a single person but at least two persons must allow the job to run are sufficient as a quick fix. They also mention changing the future architecture to automate such maintenance jobs and to make data recovery faster. So it is indeed a mix of process and technical measures, much as the problem was caused by more of a process problem rather than a technical problem.

Finally I don't think that the computational part of the data recovery is the time intensive one. A laptop may churn through gigabytes and gigabytes a minute, but a developer can't. In this case the more time intensive part is probably figuring out which parts of the data got lost and how to get them back, then writing the tools to automate the process, test that the data fits and then put it safely back in production. I understand that it's annoying, but theres only so many flops a human brain can run.

Sadly, it's only the postmortems. Their entire status gameplan involves being very vague.