Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

news.gandi.net

Post reply on HN
Print view View original

61–70 of 83 posts

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#61

Post by bradly »

bpizzi wrote: post #46
bradly wrote: post #42

Earlier quoted context omitted.

They called snapshots backups in their web interface when viewing snapshots. From their docs: > "Snapshots allow you to create a backup copy of a volume" https://pbs.twimg.com/media/EN2UZ6TX4AAMe-H?format=png&name=... They are doing a lot of preaching about backups when failing to do internal backups (not customer facing backups) of their own products.

As it's said in the postmortem, they are agreeing that they should have stated in a more obvious way that the backups availability was not contractually assured.

The postmortem says "we don’t provide a backup product for customers" while the docs describe the snapshots as a backup (see screenshot from my higher level comment). This is the disconnect for me that I'm sure is causing a lot of the frustration they are hearing from customers. They are not accepting that they sold the snapshots as a backup and this is disappointing in a postmortem where users are looking for empathy, acknowledgement, and a path forward.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#62

Post by rsync »

zaroth wrote: post #2

I’m no ZFS expert, but it must have been incredibly stressful, if not mildly terrifying, going that far down the rabbit hole with customer data on the line. I have a bad feeling someone is going to read their write up and tweet at them, “Why didn’t you use -xyz switch, it fixes exactly this issue in 12 seconds”. Indeed it appears that the option they needed existed, but only in a later version of ZFS than they were r…

"Indeed it appears that the option they needed existed, but only in a later version of ZFS than they were running, and part of the fix was moving the broken array to a system that could run a newer version of ZFS, which apparently was itself not trivial."

I have not read this post-mortem yet, but I can attest that this is a viable strategy.

As many know, rsync.net is built entirely on ZFS.

While we have never come close to a blown array (we use extremely conservatively configured raidz3 vdevs) what we have seen are weird corner cases where suddenly a 'zfs destroy' or even a common 'rm' deletion of hundreds of millions of files will either take forever (years) or will halt the (FreeBSD) system.

In one of these cases, after several days of degraded performance and intermittent outages, we did an alternate boot to a newer FreeBSD version with a newer, production, release version of ZFS, and the operation completed in a timely and graceful manner.

---

What we continue to learn, decade after decade, from UFS2 through to ZFS, is that extremely simple infrastructure configuration is resilient and fails in predictable and boring ways.

We could gain so much "efficiency" and save a lot of money if we did common sense things like bridge zpools across multiple JBODs or run larger vdevs, etc. - but then we'd find ourselves with fascinating failures instead of boring ones.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#63

Post by rsync »

ZeroCool2u wrote: post #6

This is basically the stuff of nightmares. You can't really fault them for the zfs version being so old the feature they needed wasn't yet implemented, because the machine was literally part of the last batch to be upgraded. The root cause is just some random hardware failure that can't be anticipated. Just bad luck. Beyond radically changing how their core infrastructure works, doesn't seem like there was a lot they…

"Beyond radically changing how their core infrastructure works, doesn't seem like there was a lot they could have done to prevent this."

If only there were a cloud storage provider that you could 'zfs send', over SSH, to ...

If only ...

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#64

Post by kbr2000 »

doublerabbit wrote: post #25
toomuchtodo wrote: post #19

Earlier quoted context omitted.

No offsite backups? Backblaze B2 is exceedingly cheap for example.

The animation studio I worked for had almost a petabyte of data. It may be cheap to buy the storage but transferring is costly. It's very easy to saturate a MPLS circuit with data, even rSync on a 10Gbit internal connection takes a long while. Really Gandi should of had backups from day one. If your hosting data you should always have backups ready and tested on day one.

rsync went quite fine while transferring data (in the same situation as you describe), when taken care of some important bottlenecks (not running it over SSH, disabling compression on files that don't compress well, disabling full checksums, TCP sockopts, ...)

what it might leave you hanging with for a long time is before an actual transfer, while it builds and compares the lists on both sending and receiving side, when you have big filesystems (hundreds of millions of files).

if you have a strategy to select beforehand which files to transfer (for example from a DB which tracks what has been created or changed, direct from worker or production input) you have a good headstart and can minimize rsync on complete filesystems -- and rather run it on a selection, which is tiny compared to the complete project(s) most of the time.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#65

Post by speeder »

kaizendad wrote: post #59

It's interesting -- they ID the actual cause of the problem up top, and then just zip right past it. The problem wasn't the hardware failure, or the lack of backups, it was that customers expected them to have backups . Gandi goes into some detail on the recovery process and on ways to fix the issue in the future. But, apart from some hand-waving, they don't have any specifics about how they'll communicate expectatio…

People posted an excerpt of their manual, it explicitly called "snapshots" a "backup" and customers were surprised when they asked them to restore the snapshots and only now they explained the snapshots were not backups...

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#66

Post by vel0city »

tmikaeld wrote: post #60
iicc wrote: post #32

> We think it may be due to a hardware problem linked to the server RAM. Are they using ECC RAM?

Sounds like they didn't and the metadata logs got corrupted.. This does also mean that other data would be corrupted too, running ZFS without ECC RAM is frequently warned against.

Running any resilient storage system without ECC RAM is warned against, people just really make a big deal about it with ZFS. If your data in RAM is corrupted before it makes it to the hard drive, pretty much any file system is going to write corrupted data to the drive.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#68

Post by perlgeek »

As a postmortem, this does not inspire confidence. It's a very technical piece, but doesn't even try to take a customer's perspective.

If you want to learn from such an outage, you have to do a fault analysis that leads to parameters you can can control.

Sure, there can be faulty hardware and software, but you are the ones selecting and running and monitoring them.

If recovery takes ages, you might want to practice recovery and improve your tooling.

And so on.

Blaming ZFS and faulty hardware and old software all cries "we didn't do anything wrong", so no improvements in sight.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#69

Post by kaizendad »

speeder wrote: post #65
kaizendad wrote: post #59

It's interesting -- they ID the actual cause of the problem up top, and then just zip right past it. The problem wasn't the hardware failure, or the lack of backups, it was that customers expected them to have backups . Gandi goes into some detail on the recovery process and on ways to fix the issue in the future. But, apart from some hand-waving, they don't have any specifics about how they'll communicate expectatio…

People posted an excerpt of their manual, it explicitly called "snapshots" a "backup" and customers were surprised when they asked them to restore the snapshots and only now they explained the snapshots were not backups...

Absolutely! And yet no explicit planning for how they'd reword the manual in the future. That is the root cause of the real problem here -- not that there weren't backups, but that there weren't and the manual said that there were, and there's no plan to fix that!

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#70

Post by the8472 »

vahomu wrote: post #27

> As disks are read at 3M/s, we estimate the duration of the operation to be up to 370 hours. Am I reading this right? This works out to just ~3.8TiB. So much drama over, basically, one HDD worth of data?

If they had some spare SSD capacity lying around they could have done a linear copy of the HDD to SSD and then done the import, that could have sped up the random-access scans.
Post reply on HN