Live data from Hacker News

Gandi loses data, customers told to use their own backups

status.gandi.net

121–130 of 389 posts

Re: Gandi loses data, customers told to use their own backups

#121

Hey Guys sorry for being a philistine about this but does that mean we have lost our domain name and how can we migrate it to another hosting platform? Cheers

If you only had a domain you're likely not affected (metadata). If you also have a website hosted there (data), you may be.

Re: Gandi loses data, customers told to use their own backups

#122
post #114
post #84

Earlier quoted context omitted.

It can be, it just doesn't have to be Let's say a typical admin of a small shop wants to backup his postgres database. The first thing he'll use is probably pg_dumpall which he'll output to a storage. No replication involved. The backup is just a bunch of sql statements to recover the last known state of the database. It's a different kind of format however, which -by definition- isn't a replica anymore. (And this pr…

> It's a different kind of format however, which -by definition- isn't a replica anymore. All non-trivial replication has to cross machine boundaries. To transmit to another machine, you have to use a serial format since there are no pointers on the wire. So insisting that a replica must be the same format prohibits the concept of replication in practice.

So, that admin dumped his database but didn't know that the data was using a custom locale which makes recovery troublesome.

How is it a copy if it can't recover the original in some cases?

We're not talking about a compressed archive here, it's a (incomplete) step-by-step instruction to recreate the data. If anything out of norm happens, it's gonna fail-possibly silently.

Re: Gandi loses data, customers told to use their own backups

#123

Whoops, so long with the "no bullshit" policy. I stopped using them a while ago but for a different reason. I used to use their website to check availability/whois for domains that I was interested in buying. If it was available I didn't buy it at the time but until I finished the website/app whatever I was going to put there, this took me a few months obviously. It happened to me that when I was finally ready the do…

Domain registrars giving away domains to squatters when people search for them is a time honored practice. The advice I've seen is to not search for the domain first, just register it outright from the start. The domain registration business is run with all of the integrity and customer focus of TicketMaster.

Really? I may have just found myself a new hobby. Search for incredibly unique (and worthless to me) domains to see if I can get people to squat on them. Heck, it could be a game ... I could get all my friends to make bingo boards ... or maybe see if I can think of some scrabble like rules.

Re: Gandi loses data, customers told to use their own backups

#124
post #19

Earlier quoted context omitted.

You really shouldn't trust anyone hosting your data. Always have backups!

This becomes very difficult as your data grows. If you live in AWS world, imagine periodic snapshotting from EBS, S3, RDS(and other data stores), EFS etc. For most people a different DC of the same cloud provider should be enough. If you have to put this into a different cloud provider it is a big cost drain and difficult to manage let alone if you want to have your own physical backups.

AWS has tools around this (lifecycle manager) that you can easily leverage for simple site backups. Or you can roll your own, honestly it is not that hard to take rolling snapshots.

Obviously hosting providers do not make it easy to extract your data because that's their vendor lock.

Re: Gandi loses data, customers told to use their own backups

#125
post #14

Earlier quoted context omitted.

No you don't. While agree everyone should have their own backups, you should expect your hosting company to properly replicate and backup their datacenters.

AWS recently had data loss https://news.ycombinator.com/item?id=20875489

I'd be interested in more on this claim.

- Was this mostly a power loss or a data loss?

- If data loss, did this affect EBS (which has had a claimed annual failure rate of 0.2% - 0.5% or so if I remember) or S3 (much lower failure rate). Remember, EBS WILL have volumes go bad - that's in the docs, they recommend snapshots, aws backup manager etc if you need higher durability.

Re: Gandi loses data, customers told to use their own backups

#126

Earlier quoted context omitted.

This is god damn unbelievable "Andrea, sorry about that and the incident. If we led you to believe that you had nothing to do on your side when warned multiple times to make your back ups, then we'll have to make it clearer, and stop assuming that it's an industry wide knowledge."

Big words for a company that's in trouble for not backing up data themselves.

[deleted]

Re: Gandi loses data, customers told to use their own backups

#127

Earlier quoted context omitted.

They put a rude word on their homepage, that makes them edgy and cool and anti-corporation!

"No bullshit" is up there on my corpro-speak charts right along with "synergy" and "innovation". Everyone's website says they're "no bullshit". It's all bullshit.

"No bullshit" works when it's an SME talking, but once a company reaches a certain size then all bets are off

Re: Gandi loses data, customers told to use their own backups

#128
post #4

Earlier quoted context omitted.

A promo code in exchange of your data loss. What a bargain!

“Please keep trusting us to host your data”

"...marginally more than rolling your own or another cloud provider."

And to "trust marginally more" simply means:

    gandi_cost_per_month + P(gandi_fails_per_month) * cost_recovery
    

Re: Gandi loses data, customers told to use their own backups

#129
post #103
post #18

Earlier quoted context omitted.

In the last few years, I have seen many people confuse replication with backups. People see them as the same thing, but they really aren't. Even with snapshots, if the devices are the same, they might have the same firmware bug, etc.

Just to further explore that a bit, would you say replication adds independent copies for failures of media, while backup adds copies made by independent software against failures of process / software / media.

Replication increases risk of data loss when implemented incorrectly, because added resources increase the probability of bit errors. This applies to both replicated disks (RAID) and servers. Replicated servers must use ECC memory as well as checksum blocks and periodically scrub data to ensure integrity (e.g. what ZFS does for you). If they don't then a bit error corrupts the data on all servers, because you have no way of know which copies are pristine or how to piece together pristine parts.

Re: Gandi loses data, customers told to use their own backups

#130
The key question is, did Gandi offer and explicit backup service for your data on their plans? I just had a look and I don't see this being offered.

As a former hosting engineer, at the risk of pissing on everyone's outrage parade, but unless an explicit guarantee of a backup is included in your plan's contract, or you can pay for backups as a bolt-on, then if you've lost data it's your fault for not planning for this scenario.

And I mean proper backups where you get, for example, twenty eight days of hourly backups and you can pick a specific version of file to recover in that 28 period. And where those backups are stored on different hardware or off-site. We offered this as a bolt-on (in-site and off-site). Tt was 20 quid a year for in-site, the off-site was a bit more. But a great many customers chose not to pay for this add-on, even despite the great big red bold warning text explaining that unless they paid for this add-on we made no guarantees about the permanence of their data in the event of a storage problem. Guess what....

Now that's not saying we didn't take snapshots of the hosting environment, but they were for internal use and to allow us to recover quickly in the event of something unexpected going wrong, but now and again stuff breaks.

Sure, it's unfortunate some lump of storage hardware has failed and whatever mirrors they may have had have been taken out as well. They possibly could have done better but shit happens sometime.

You shouldn't rely on an "implied backup" from your service provider, if you want that then you're going to be paying a shedload more for hosting your Wordpress and Woocommerce site. It's up to you to make sure absolutely sure your data is safe if it's critical to the day-to-day running of your business.

Edit: ok, so this is tucked away in their docs (thanks to itake below):

https://docs.gandi.net/en/simple_hosting/common_operations/s...

But it does say:

> Snapshots do not make a backup of your databases. If you would like to perform a backup of your databases, we recommend you perform an export, or launch a dump script via crontab.

The bottom line...is it guaranteed in your contract? Always check. And as per my follow up comment, those plan prices are are just too cheap for that facility to be taken seriously for business continuity. They're a convenience to quickly recover a version of a file, not a serious backup.

Post reply on HN