Live data from Hacker News

Outage post-mortem

tech.dropbox.com

1–10 of 44 posts

Re: Outage post-mortem

#4
I found it hard to work out where to get the most up to date information on the outage. I checked the blog, but the last message was their New Year message. In the app and on the main website (mobile version) I couldn't see anything...

Glad I read HN otherwise I don't know how I would have come across this information. =)

Re: Outage post-mortem

#6
This is, sadly, not a great post-mortem. They missed an opportunity for goodwill. I don't feel more confident in their level of understanding or ability to remediate the problems that led to it after having read it. I know they have an excellent engineering and operations staff -- this post-mortem doesn't reinforce that, though.

A few of the things that jumped out at me after one reading:

1. The apology is the next to last sentence. That's burying the lede. I'd like to see that far earlier, in the first two to three sentences.

2. The tone is overly clinical and lacks humanity. I suspect they felt that it made them sound more authoritative and in control, but instead it comes off somewhat robotic.

3. There's a mixture of too little and too much technical detail. It feels like they couldn't decide who the audience was. There were technical tidbits thrown out without any elaboration that lead to more questions than answers.

4. The remediations sound pretty weak. There's no discussion of the human factors like how the recovery process went, how this issue was missed in testing, or what changes if any they think they should make to their incident response process. At the very least I'd expect to see some remediation around their during-outage communication process since it has pretty universally been considered to be poor.

It's not the worst post-mortem I've read, but they missed a few chances to reassure customers.

Re: Outage post-mortem

#7
post #3

I thought Percona's Xtrabackup already supported parallelized recovery using binary logs: http://www.percona.com/doc/percona-xtrabackup/2.1/ (Though I'll admit I've never tried to do so)

This is the same problem as to why slaves were single-threaded for so long:

Statement based replication (default) is tricky to apply in parallel, since you can't easily figure out ordering dependencies.

In MySQL 5.6 replication is now parallel per-schema, and in MySQL 5.7 it will be parallel intra-schema.

Re: Outage post-mortem

#8
post #6

This is, sadly, not a great post-mortem. They missed an opportunity for goodwill. I don't feel more confident in their level of understanding or ability to remediate the problems that led to it after having read it. I know they have an excellent engineering and operations staff -- this post-mortem doesn't reinforce that, though. A few of the things that jumped out at me after one reading: 1. The apology is the next t…

As I've said before, one blog post does not represent that team when it's written by someone tasked with the job of communicating with a wide variety of customers. My mom could give two hoots about details. She wants to know why her 'spinny drobox thing' keep spinning and should she upgrade or something. I deliver that news to her. This blog post delivers it to people who don't understand as well as most of us but better than my mom.

What would be an AWSOME idea is if Dropbox did a meetup to go through the gory details for us nerds. Now that would rock.

Kudos the the Dropbox team for working through the weekend fixing stuff. I spent the better part of the weekend nursing a barely 2 year old dying Apple 27" Cinema Display back to life by disassembling it several times. Kept thinking to myself that I sure as hell was glad it wasn't me over at Dropbox HQ working on doing recovery instead.

Edit: I agree with your plea for emotion in the post. It could ease things a bit.

Re: Outage post-mortem

#9
This incidence clearly shows the fragile nature of public cloud. All your data is just one Shell command away. Glad Dropbox is not a backup service.

Re: Outage post-mortem

#10
post #4

I found it hard to work out where to get the most up to date information on the outage. I checked the blog, but the last message was their New Year message. In the app and on the main website (mobile version) I couldn't see anything... Glad I read HN otherwise I don't know how I would have come across this information. =)

Would be nice to have a service who stepped in when shit like this happened. I'd pay good money to have a tiger team appear out of thin air when the shit hit the fan.
Post reply on HN