This is, sadly, not a great post-mortem. They missed an opportunity for goodwill. I don't feel more confident in their level of understanding or ability to remediate the problems that led to it after having read it. I know they have an excellent engineering and operations staff -- this post-mortem doesn't reinforce that, though. A few of the things that jumped out at me after one reading: 1. The apology is the next t…
Outage post-mortem
41–44 of 44 posts
Re: Outage post-mortem
#42Humans have a lot going for them. They can think continuously and dynamically. They can change their instructions at a whim. They can provide custom solutions immediately. And they aren't limited to one way to solve a problem.
When you have to perform a bunch of complicated changes in bulk, you might think automating it would be the best way to ensure a uniform delivery of your changes. But when a single thing is different about one environment, everything is fucked. The only way to ensure a bunch of sensitive changes go off without a hitch is to make it a manual process, even if you have to supplement it with some automated processes along the way.
In this case, Dropbox allowed their site's reliability to be dictated by the automated maintenance of production servers. It's always dangerous to make changes on a production server. But what makes it worse here is that they relied on a script to make sure everything happened perfectly, and didn't double-check the results before putting it back into production.
They didn't even back up the old data in case they needed to quickly revert, which should be a basic requirement of any production change! This isn't even disaster recovery, this is production deployment 101. How they allowed this upgrade to affect the production site is just crazy to me.
Re: Outage post-mortem
#43Earlier quoted context omitted.
> Wouldn't you have wanted DropBox to let you know something was going on? This is such a hard thing to balance. I hate nagging notifications. Dropbox makes it really easy to see what the service status is. I glance at my menu bar/system tray icon, and look for the green check. If I don't see the green check, I know my docs haven't synced.
This is a glaring case of rose tinted glasses! Of course you'd want to know you were working on out of date src/document/etc...
Green check - good to go! Blue cycle - syncing Blank icon - no connection
Not wanting a nag is not rose tinted glasses. If there is an improvement to be made, it would be in the last icon. Blank doesn't exactly scream "we're down". If the Dropbox client can't get a connection to the service, but it can see that a network link is available, it should give some indication that it is not connected, like some manner of red indicator.
Re: Outage post-mortem
#44The great thing about DropBox is that I didn't even notice, despite using my files across the outage.