Outage post-mortem
tech.dropbox.com
Outage post-mortem
1–10 of 44 posts
Re: Outage post-mortem
#2Re: Outage post-mortem
#3Re: Outage post-mortem
#4Glad I read HN otherwise I don't know how I would have come across this information. =)
Re: Outage post-mortem
#5Re: Outage post-mortem
#6A few of the things that jumped out at me after one reading:
1. The apology is the next to last sentence. That's burying the lede. I'd like to see that far earlier, in the first two to three sentences.
2. The tone is overly clinical and lacks humanity. I suspect they felt that it made them sound more authoritative and in control, but instead it comes off somewhat robotic.
3. There's a mixture of too little and too much technical detail. It feels like they couldn't decide who the audience was. There were technical tidbits thrown out without any elaboration that lead to more questions than answers.
4. The remediations sound pretty weak. There's no discussion of the human factors like how the recovery process went, how this issue was missed in testing, or what changes if any they think they should make to their incident response process. At the very least I'd expect to see some remediation around their during-outage communication process since it has pretty universally been considered to be poor.
It's not the worst post-mortem I've read, but they missed a few chances to reassure customers.
Re: Outage post-mortem
#7I thought Percona's Xtrabackup already supported parallelized recovery using binary logs: http://www.percona.com/doc/percona-xtrabackup/2.1/ (Though I'll admit I've never tried to do so)
Statement based replication (default) is tricky to apply in parallel, since you can't easily figure out ordering dependencies.
In MySQL 5.6 replication is now parallel per-schema, and in MySQL 5.7 it will be parallel intra-schema.
Re: Outage post-mortem
#8This is, sadly, not a great post-mortem. They missed an opportunity for goodwill. I don't feel more confident in their level of understanding or ability to remediate the problems that led to it after having read it. I know they have an excellent engineering and operations staff -- this post-mortem doesn't reinforce that, though. A few of the things that jumped out at me after one reading: 1. The apology is the next t…
What would be an AWSOME idea is if Dropbox did a meetup to go through the gory details for us nerds. Now that would rock.
Kudos the the Dropbox team for working through the weekend fixing stuff. I spent the better part of the weekend nursing a barely 2 year old dying Apple 27" Cinema Display back to life by disassembling it several times. Kept thinking to myself that I sure as hell was glad it wasn't me over at Dropbox HQ working on doing recovery instead.
Edit: I agree with your plea for emotion in the post. It could ease things a bit.
Re: Outage post-mortem
#9Re: Outage post-mortem
#10I found it hard to work out where to get the most up to date information on the outage. I checked the blog, but the last message was their New Year message. In the app and on the main website (mobile version) I couldn't see anything... Glad I read HN otherwise I don't know how I would have come across this information. =)