Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

I’m no ZFS expert, but it must have been incredibly stressful, if not mildly terrifying, going that far down the rabbit hole with customer data on the line.

I have a bad feeling someone is going to read their write up and tweet at them, “Why didn’t you use -xyz switch, it fixes exactly this issue in 12 seconds”.

Indeed it appears that the option they needed existed, but only in a later version of ZFS than they were running, and part of the fix was moving the broken array to a system that could run a newer version of ZFS, which apparently was itself not trivial.

And now their postmortem blog post is down with a 503 error. Doesn’t exactly fill me with confidence about their abilities.

This is basically the stuff of nightmares.

You can't really fault them for the zfs version being so old the feature they needed wasn't yet implemented, because the machine was literally part of the last batch to be upgraded. The root cause is just some random hardware failure that can't be anticipated.

Just bad luck. Beyond radically changing how their core infrastructure works, doesn't seem like there was a lot they could have done to prevent this. Kudos for releasing the post mortem though, at least they've been fairly honest and direct about it.

Their services otherwise run flawlessly for me. I appreciate the transparency.

Would it be possible to backup the precious metadata separately to mitigate the issue?