21–30 of 40 posts
1. Demonstrate understanding of the event.
2. Communicate what steps are being taken to reduce the likelihood a similar event in the future.
3. Apologize.
Not necessarily in that order. Well done.
Earlier quoted context omitted.
> People can't own up to the fact that customers don't really care if it's Amazon AWS or Heroku or some other platform that failed -- if your service is down, its your fault! I don't, necessarily, have a problem with saying, without animosity, "we're down because our hosting provider is down". For many small businesses, being down when your hosting provider is down is an acceptable tradeoff -- your business simply is…
"people who were behaving as though they were fine with that tradeoff" I'm not sure who is to blame at this point -- is it the fault of the individual for not understanding uptime? is it the fault of the service for not articulating clearly what it means? Is it the fault of the industry for inculcating an unjustified sense of entitlement?
I'm inclined to lean towards the third option. It ought to be obvious that there is no such thing as a perfect host, that in the limit your probability of being down at some point goes to 100% as your time hosted with a company increases.
There's just a lot of entitlement and sloppy thinking out there. People believe that their $30 a month hosting ought to be magically invulnerable in ways it by the simple laws of physics cannot be. If you didn't explicitly choose to have multiple geographically separate hosts with a fail-over plan between them, then you were at the very least implicitly accepting that you were going to be down some of the time when (not if) your only host goes down. That's a fine and ok thing to choose, but I don't think you get to be mad at your host because you didn't clue in to the choice you were making.
It's amazing that the team worked through what is likely the least fun night of the year to be working to fix this issue.
I wonder who gets stuck working those shifts -- would it just be following the normal schedule, in the name of fairness? Do more senior people get to claim the day of vacation? Do parents get priority over single people?
It's amazing that the team worked through what is likely the least fun night of the year to be working to fix this issue.
You could be standing next to Jeff Bezos, putting books in boxes. The irony for me was that it was one of the few times I could expect to see him, despite my (small slice of an) office being two doors down from his.
Now that there are multiple, largely automated, DCs, I'm not sure if this still happens. Amazon isn't really "the little guy" anymore, at least not in the retail sense.
12:24 on 12/24. Interesting coincidence. What are the odds?
And no, those are not the odds.
12:24 on 12/24. Interesting coincidence. What are the odds?
12:24 on 12/24. Interesting coincidence. What are the odds?
Two provisos: first, it's really late so there's probably something wrong with the math up there; second, that wouldn't count other significant patterns (like, 01/02 03:04 or something) because that's kinda difficult to quantify.
1) When was the problem detected? 2) How was the problem detected: automated monitoring, operator vigilance, or customer complaints? 3) When were customers first notified of the outage?
This post-mortem starts out with the root cause, but it is likely that was not known until after the outage had ended, so it's important to know how much time passed between the root cause event and when the on-call engineer(s) started treating the outage as such.
I'd also be interested in hearing about technical measures they will use to prevent this in the future. It sounds like their mitigation measures are all social, not technical. For example, do their configuration tools warn them when they are about the push out a huge delta? Do they have a review process that forces two people to look at the change before it's put into action? Would these have prevented the outage?
Finally, I think it's appalling (but not surprising) that it took many hours to revert their configs. It sounds like the tools they are using are way too slow. Any laptop can do an enormous amount of work in an hour; the computing resources available to Amazon can perform truly insane feats in one minute. Slow configuration tools are easy to write but prolong outages. They should be designed to the same standards as production applications.
It's amazing that the team worked through what is likely the least fun night of the year to be working to fix this issue.
I wonder who gets stuck working those shifts -- would it just be following the normal schedule, in the name of fairness? Do more senior people get to claim the day of vacation? Do parents get priority over single people?