Previously: https://news.ycombinator.com/item?id=22001822
Postmortem of the failure of one hosting storage unit on Jan. 8, 2020
41–50 of 83 posts
Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020
#42They don't say they're sorry, because they're not. Instead they minimize their actions by: 1) stating how few customers customers were affected, 2) how it's not really their fault because it was a hardware error, 3) it's not really their fault because they had already planned to upgrade the server, 4) it's not really their fault the restore procedure took so long because they had to make backups first, 5) the restore…
Well, it's a technical post-portem, not a love letter. I'm not affiliated to Gandi in any way, but I find the finger-pointing a bit too pedantic. > The take-away here is clear: don't trust Gandi with anything you care about. The take-away is not this one. Its: backup anything you care about.
They called snapshots backups in their web interface when viewing snapshots. From their docs:
> "Snapshots allow you to create a backup copy of a volume"
https://pbs.twimg.com/media/EN2UZ6TX4AAMe-H?format=png&name=...
They are doing a lot of preaching about backups when failing to do internal backups (not customer facing backups) of their own products.
Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020
#43Earlier quoted context omitted.
There was one, obvious thing they could have done. Backups. ZFS even makes it easy. They state they have triple redundancy on their servers. Pull 1/2 of the drives and have backups. ZFS supports streaming snapshots (maybe not on this particularly old system). It sounds like they have multiple ZFS servers per datacenter, so given 2 servers, they could use 50% of the storage on each nodes as a backup of the other node.…
They do mention in the postmortem that they explicitly do not provide backups, and say so on their product page, but perhaps that it could be better communicated to customers. Designing a really robust system to failures like this is a very difficult problem. You can see this in the complexity of systems like S3 and Google's Colossus[1]. Colossus in particular is probably one of Google's single greatest competitive a…
Do not provide backups and don't have backups aren't the same thing. If you have triply redundant local disks, you can probably afford take half of them and use them as backups for other systems and achieve better availability results (I'm assuming it's not triply redundant for performance).
While I did say S3, what I was really thinking about was Ceph. I don't think it's a silver bullet (almost certainly way more maintenance than a bunch of ZFS nodes), but if you're big enough to have multiple storage nodes with 100's of customers each (and again, triply redundant disks), then you could have built around the eventual failure of a node with what you already have. I'm not expecting them to hit S3's 11 9 availability, just taking a glance at what they have said about their design and proposing that basic changes to how they allocate what they already have would have avoided their problem in the first place.
I don't know what their exact situation looks like, or how they got into this situation. I see a post-mortem that says they spent 5 days trying desperately to recover customer data because they don't have backups, and they're not going to change anything about how they do things to eliminate the problem, even though it appears they have the raw storage capacity to have a backup. A sister comment says that brand damage, customer costs and recovery costs are just hypotheticals. They were, right up until this incident. Hopefully their internal postmortem has more details about what the costs were.
Clearly if they're trying to recover the customer data, it was important enough to the business to do so, and maybe it's time to re-evaluate 'no backups'.
Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020
#44>But contractually, we don’t provide a backup product for customers. That may have not been explained clearly enough in our V5 documentation. If you have a single point of failure for data and "snapshots" then you should explain that very clearly to customers. Moreover, as I understand it, competitors like AWS do not have such a single point of failure (ie: EBS Snapshots are on S3 and not EBS) so using the same termi…
Are S3 and EBS not hosted on the same underlying storage subsystems?
Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020
#45Earlier quoted context omitted.
The linked twitter thread explicitly mentions data loss: > Hi Andrea. It is confirmed we have lost data and we are terribly sorry for that. However, please note that what happenend[sp] could happen to any web host. Customers that were forced to migrate to a different webhost had to restore from whatever backups they had, and they lost data for sure. Even if Gandi ultimately recovered everything (and it's not complete…
OK, that makes it even worse then. The postmortem linked above definitely says: > We managed to restore the data and bring services back online the morning of January 13. Is that wrong? It's bad to lose data, it's even worse to tell people you didn't lose data in one place when you did, and tell them you did lose data in another.
There was confusion with this. Originally they thought they lost all data, which is why a lot of people went crazy at them via Twitter, they later said there now might be a chance to recover data - luckily they ended up finding a way to recover it.
Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020
#46Earlier quoted context omitted.
Well, it's a technical post-portem, not a love letter. I'm not affiliated to Gandi in any way, but I find the finger-pointing a bit too pedantic. > The take-away here is clear: don't trust Gandi with anything you care about. The take-away is not this one. Its: backup anything you care about.
They called snapshots backups in their web interface when viewing snapshots. From their docs: > "Snapshots allow you to create a backup copy of a volume" https://pbs.twimg.com/media/EN2UZ6TX4AAMe-H?format=png&name=... They are doing a lot of preaching about backups when failing to do internal backups (not customer facing backups) of their own products.
As it's said in the postmortem, they are agreeing that they should have stated in a more obvious way that the backups availability was not contractually assured.
Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020
#47They don't say they're sorry, because they're not. Instead they minimize their actions by: 1) stating how few customers customers were affected, 2) how it's not really their fault because it was a hardware error, 3) it's not really their fault because they had already planned to upgrade the server, 4) it's not really their fault the restore procedure took so long because they had to make backups first, 5) the restore…
No data was lost though, true? I don't know if I expect a postmortem to say "sorry", and I think you are being needlessly harsh. But I agree this level of service doesn't seem up to current best in class. Like Amazon etc. (Which of course still have unexpeted outages very occasionally, although a 5 day time to recovery would certainly be... unusual). But this partially shows how much expectations/standards have raise…
> No data was lost though, true?
What about any data that would have accumulated in those 5 days? This was storage for their IAAS and PAAS products, so anyone using those lost access for 5 days?
Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020
#48> As disks are read at 3M/s, we estimate the duration of the operation to be up to 370 hours. Am I reading this right? This works out to just ~3.8TiB. So much drama over, basically, one HDD worth of data?
[deleted]
Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020
#49> As disks are read at 3M/s, we estimate the duration of the operation to be up to 370 hours. Am I reading this right? This works out to just ~3.8TiB. So much drama over, basically, one HDD worth of data?
Absolutely pathetic on their end.
Re: Postmortem of the failure of one hosting storage unit on Jan. 8, 2020
#50> As disks are read at 3M/s, we estimate the duration of the operation to be up to 370 hours. Am I reading this right? This works out to just ~3.8TiB. So much drama over, basically, one HDD worth of data?
They said that in the end only 414 users were affected, and it was their simple hosting package. Honestly, I’m almost surprised it was that much.