Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

news.gandi.net

Post reply on HN
Print view View original

71–80 of 83 posts

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#71

Post by RantyDave »

generalpass wrote: post #18
escardin wrote: post #14

Earlier quoted context omitted.

There was one, obvious thing they could have done. Backups. ZFS even makes it easy. They state they have triple redundancy on their servers. Pull 1/2 of the drives and have backups. ZFS supports streaming snapshots (maybe not on this particularly old system). It sounds like they have multiple ZFS servers per datacenter, so given 2 servers, they could use 50% of the storage on each nodes as a backup of the other node.…

> How much did five days of panic cost them? Their customers? Their brand? Intangibles. Whenever you go to talk to management or even co-workers about this stuff, they look at you like you are crazy. I think it is just human nature to not even think that something could go wrong, let alone make decisions based on this.

Oh, I think they recognise the damage to their brand.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#72

Post by RantyDave »

tmikaeld wrote: post #9

This actually makes me glad to use ZFS (FreeBSD and ZOL) on all servers, a broken RAID on a different filesystem could have meant complete data-loss. From all of the cases I've read where people where not idiots (Not using snapshots and overwriting a dataset..), it's by far the safest filesystem I've seen during my 12 years working with it and I've yet to loose a single file. Sure, performance can suffer and RAM is p…

But would this replicate the broken metadata? And exactly how do they think it got there in the first place? TBH this sounds like a problem with an ancient version of Solaris that's been enhanced by a relatively small company and it's finally just bitten them.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#73

Post by RantyDave »

gdm85 wrote: post #10

Would it be possible to backup the precious metadata separately to mitigate the issue?

I don't think it's a problem of the data corrupting while on disk, I think the problem has occurred in ram and then been written to disk.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#74

Post by generalpass »

RantyDave wrote: post #71
generalpass wrote: post #18

Earlier quoted context omitted.

> How much did five days of panic cost them? Their customers? Their brand? Intangibles. Whenever you go to talk to management or even co-workers about this stuff, they look at you like you are crazy. I think it is just human nature to not even think that something could go wrong, let alone make decisions based on this.

Oh, I think they recognise the damage to their brand.

> Oh, I think they recognise the damage to their brand.

Today, yes. Two years ago?

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#76

Post by _nalply »

robbyt wrote: post #29
_nalply wrote: post #13

Just bad luck. A different story: I had a cheap dedicated host in Atlanta. Their failure was epic. You get what you pay for. An old highrise is filled with ten thousands of old second hand server blades, floors by floors of equpiment prolifically producing waste heat. A sure recipe for disaster? Sure! A wrongly installed fuse at one phase in the building made one phase burn out too early. I saw a picture of an archeo…

Not bad luck, bad design. Same for Gandi, and for the situation you just described. Drive failures and HVAC failures due to bad power are not "black swawn" events. These are very common problems for DCs, and a good design takes these problems into account. However, a "bad" design is cheap, and hopefully the savings is passed to the customer.

Hopefully... My dedicated hosting was very cheap, I paid $20 a month.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#77

Post by tmikaeld »

vel0city wrote: post #66
tmikaeld wrote: post #60

Earlier quoted context omitted.

Sounds like they didn't and the metadata logs got corrupted.. This does also mean that other data would be corrupted too, running ZFS without ECC RAM is frequently warned against.

Running any resilient storage system without ECC RAM is warned against, people just really make a big deal about it with ZFS. If your data in RAM is corrupted before it makes it to the hard drive, pretty much any file system is going to write corrupted data to the drive.

Indeed, ECC should be _default_ these days!

It becomes even worse when RAM Is usually only tested when the computer is built.

I've had several cases where RAM becomes faulty a couple of years down the road.

Recently I had a very weird case of two stick of four went bad due to only moving the computer from one corner to the next without even opening the case.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#78

Post by jotm »

doublerabbit wrote: post #25
toomuchtodo wrote: post #19

Earlier quoted context omitted.

No offsite backups? Backblaze B2 is exceedingly cheap for example.

The animation studio I worked for had almost a petabyte of data. It may be cheap to buy the storage but transferring is costly. It's very easy to saturate a MPLS circuit with data, even rSync on a 10Gbit internal connection takes a long while. Really Gandi should of had backups from day one. If your hosting data you should always have backups ready and tested on day one.

Hey, good business idea: backup storage in vans! :)

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#79

Post by halfeatenpie »

_nalply wrote: post #76
robbyt wrote: post #29

Earlier quoted context omitted.

Not bad luck, bad design. Same for Gandi, and for the situation you just described. Drive failures and HVAC failures due to bad power are not "black swawn" events. These are very common problems for DCs, and a good design takes these problems into account. However, a "bad" design is cheap, and hopefully the savings is passed to the customer.

Hopefully... My dedicated hosting was very cheap, I paid $20 a month.

It's Delimiter, wasn't it? They weren't a very good hosting organization to begin with, but you get what you pay for.

Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020

#80

Post by ficklepickle »

jotm wrote: post #78
doublerabbit wrote: post #25

Earlier quoted context omitted.

The animation studio I worked for had almost a petabyte of data. It may be cheap to buy the storage but transferring is costly. It's very easy to saturate a MPLS circuit with data, even rSync on a 10Gbit internal connection takes a long while. Really Gandi should of had backups from day one. If your hosting data you should always have backups ready and tested on day one.

Hey, good business idea: backup storage in vans! :)

AWS beat you to it[0]. Not a van, but a 45 foot trailer.

[0] https://aws.amazon.com/snowmobile/

Post reply on HN