Live data from Hacker News

Outage post-mortem

tech.dropbox.com

41–44 of 44 posts

Re: Outage post-mortem

#41
post #6

This is, sadly, not a great post-mortem. They missed an opportunity for goodwill. I don't feel more confident in their level of understanding or ability to remediate the problems that led to it after having read it. I know they have an excellent engineering and operations staff -- this post-mortem doesn't reinforce that, though. A few of the things that jumped out at me after one reading: 1. The apology is the next t…

Google provided a great Incident Report / Postmortem when they had their API infrastructure outage back in May 2013. I created a screencast about how their template should be used as a model for the rest of us to follow. You can watch the screencast @ http://sysadmincasts.com/episodes/20-how-to-write-an-inciden...

Re: Outage post-mortem

#42
They completely missed the real lessons from this outage: Automation is way more fallible than human beings, and they didn't follow basic best practices to stage and test production maintenance.

Humans have a lot going for them. They can think continuously and dynamically. They can change their instructions at a whim. They can provide custom solutions immediately. And they aren't limited to one way to solve a problem.

When you have to perform a bunch of complicated changes in bulk, you might think automating it would be the best way to ensure a uniform delivery of your changes. But when a single thing is different about one environment, everything is fucked. The only way to ensure a bunch of sensitive changes go off without a hitch is to make it a manual process, even if you have to supplement it with some automated processes along the way.

In this case, Dropbox allowed their site's reliability to be dictated by the automated maintenance of production servers. It's always dangerous to make changes on a production server. But what makes it worse here is that they relied on a script to make sure everything happened perfectly, and didn't double-check the results before putting it back into production.

They didn't even back up the old data in case they needed to quickly revert, which should be a basic requirement of any production change! This isn't even disaster recovery, this is production deployment 101. How they allowed this upgrade to affect the production site is just crazy to me.

Re: Outage post-mortem

#43
post #40

Earlier quoted context omitted.

> Wouldn't you have wanted DropBox to let you know something was going on? This is such a hard thing to balance. I hate nagging notifications. Dropbox makes it really easy to see what the service status is. I glance at my menu bar/system tray icon, and look for the green check. If I don't see the green check, I know my docs haven't synced.

This is a glaring case of rose tinted glasses! Of course you'd want to know you were working on out of date src/document/etc...

I already know that. The Dropbox icon is always within sight and provides immediate feedback on the status of your Dropbox:

Green check - good to go! Blue cycle - syncing Blank icon - no connection

Not wanting a nag is not rose tinted glasses. If there is an improvement to be made, it would be in the last icon. Blank doesn't exactly scream "we're down". If the Dropbox client can't get a connection to the service, but it can see that a network link is available, it should give some indication that it is not connected, like some manner of red indicator.

Re: Outage post-mortem

#44

The great thing about DropBox is that I didn't even notice, despite using my files across the outage.

It refused to run for me insisting that I relink my machine, and then taking me to the website to do so where that then gave an error. It was only this morning that I finally managed to get things running again.
Post reply on HN