Live data from Hacker News

Gandi loses data, customers told to use their own backups

status.gandi.net

41–50 of 389 posts

Re: Gandi loses data, customers told to use their own backups

#41
Looks like their backups only consisted of in-region backups on systems that were homogeneous. Common pitfall. While technically a 3-node distributed system may provide disaster recovery from one node failing, in practice, an accidental rm -rf from an ansible script targeting all three machines, or a bug in the software that's doing the replication, will leave you without a backup plan.

If you're in such a situation, The easiest is to do filesystem level backups with something like zfs and ship the backups to a third-party system that only has write/append-only semantics (better yet, use a write-once-read-many (WORM) disk to really guarantee it.).While there will still be _some_ data loss, it'll let you recover since the last snapshot.

If you don't have zfs, a database backup that runs the db dump script and scp/sftps it to a server running as a cronjob can also be an immediate remedy while you get your shit together (and by that I mean buy yourself a product with an immaculate reputation like aurora or cockroachdb to manage the db for you)

Harder but better would be to tee the log of the changestream (all distributed systems have such a log) to a third-party system. This is ideal because if it's done synchronously it'll let you recover since the last committed transaction.

And of course, test your backups, because backups are subject to code rot as well.

Re: Gandi loses data, customers told to use their own backups

#42
post #25
post #10

> We sincerely apologize for the inconvenience that this situation has caused. This type of incident is extremely rare in the web hosting industry. Why are they speaking of the "industry" as a whole when they are to blame? It's even crazier they are not even explaining the source of the data loss and why the "replication systems" didn't help. IHMO they are trying to sweep this event under the carpet. They should inst…

Replication is not a backup as was already mentioned. A great example of this is when the KDE project almost lost all of their Git repos because they were mirroring a corrupted copy of the data. https://www.phoronix.com/scan.php?page=news_item&px=MTMzNTc

A backup is a replication of the live dataset, although, usually out of sync to be useful when the main dataset goes bad.

Re: Gandi loses data, customers told to use their own backups

#43
post #9

I understand people might be upset because they lost data, but as a sysadmin, my reaction is "ooh shit, poor guys, that must be a horrible week"... And honestly, if you don't keep data of stuff you host on a server provider like this, you kind of get what you deserve...

> And honestly, if you don't keep data of stuff you host on a server provider like this, you kind of get what you deserve...

While I agree that everyone should have their own off-site backups, this does come across as incredibly crass victim blaming.

Re: Gandi loses data, customers told to use their own backups

#44
post #27
post #13

I don't have hosting with Gandi, but I do use them for domains and DNS. I'll be considering migrating my domains from them after this. Their response to this is exceptionally poor. To say essentially "this could happen to any other web host" it nonsense. I've never had this happen with any of the providers I've used for hosting and I'd be very angry if I had just lost an entire VPS. The fact that they've lost all sna…

I use Gandi for domains & DNS too. I've never had any problems so far but I don't want any surprises... Where do you want to migrate? What is a better alternative?

Same here. Wondering what alternative there is. Heard good things about https://porkbun.com/

Re: Gandi loses data, customers told to use their own backups

#45
post #3

Earlier quoted context omitted.

Wow, for a company that boasts "no bullshit", only offering a month after destroying data and backups seems a little tone deaf Edit: in fairness, I'm not sure how exactly you would quantify such a loss anyway...

It sounds like they didn’t have any backups at all but rather relied on a active-active replication link to a secondary storage. Edit: who knows it may be related to the HPE issue. https://www.bleepingcomputer.com/news/hardware/hp-warns-that...

Hmm... I wonder what the "incident" was. If it involved something akin to an "rm -rf," then of course their replication link didn't protect them.

Re: Gandi loses data, customers told to use their own backups

#46

Earlier quoted context omitted.

I’ve always been a little confused about their cult following given their unfriendly terms — arbitrary domain cancellation based on adult material for example — which are fair terms to have if that’s their ethics but it seems at odds with the typical pro freedom expectations many people in technology hold.

They put a rude word on their homepage, that makes them edgy and cool and anti-corporation!

"No bullshit" is up there on my corpro-speak charts right along with "synergy" and "innovation".

Everyone's website says they're "no bullshit". It's all bullshit.

Re: Gandi loses data, customers told to use their own backups

#47
post #27
post #13

I don't have hosting with Gandi, but I do use them for domains and DNS. I'll be considering migrating my domains from them after this. Their response to this is exceptionally poor. To say essentially "this could happen to any other web host" it nonsense. I've never had this happen with any of the providers I've used for hosting and I'd be very angry if I had just lost an entire VPS. The fact that they've lost all sna…

I use Gandi for domains & DNS too. I've never had any problems so far but I don't want any surprises... Where do you want to migrate? What is a better alternative?

I like Cloudflare and find them to be a very good value proposition. They have a domain registrar now as well, though I haven't tried that yet. https://www.cloudflare.com/products/registrar/

Re: Gandi loses data, customers told to use their own backups

#48
post #31
post #19

Earlier quoted context omitted.

You really shouldn't trust anyone hosting your data. Always have backups!

Often times the backup provider is the hosting provider, whom you have to trust. (This extends all the way from big clouds like AWS and GCE to small providers like Linode and DO). Having an external backup can be unreasonably expensive due to ridiculous egress costs.

You can still back up to the same providers' different data center. Two data centers failing simultaneously is very unlikely.

Re: Gandi loses data, customers told to use their own backups

#49
post #28

Oof, this Twitter thread looks particularly bad, especially the response from the official Gandi account. https://twitter.com/andreaganduglia/status/12151991477012316... While I appreciate that there are real people behind these companies that are probably having a really rough time right now, the criticism that Gandi are getting as a company is justified - and if Gandi are truly a "no bullshit" company they need to…

Screenshotted in case (when) they delete it https://i.imgur.com/s3R1VVc.png

Using memes after permanently losing customer data is extremely disrespectful.

Re: Gandi loses data, customers told to use their own backups

#50
shit happens...but the way their philosophy is fine tuned makes me wonder..

Above all, "no bullshit" is our golden rule—to treat our users how we want to be treated. It's a promise to respect your rights and to level with you about our shortcomings.

https://www.gandi.net/en-US/no-bullshit

ex: https://twitter.com/andreaganduglia/status/12151991477012316... (thanks op)

We will listen to you, and be honest in our replies, even if it means you won’t always like what we say.

Post reply on HN