Live data from Hacker News

Gandi loses data, customers told to use their own backups

status.gandi.net

31–40 of 389 posts

Re: Gandi loses data, customers told to use their own backups

#31
post #19

Earlier quoted context omitted.

“Please keep trusting us to host your data”

You really shouldn't trust anyone hosting your data. Always have backups!

Often times the backup provider is the hosting provider, whom you have to trust. (This extends all the way from big clouds like AWS and GCE to small providers like Linode and DO). Having an external backup can be unreasonably expensive due to ridiculous egress costs.

Re: Gandi loses data, customers told to use their own backups

#32
post #3

Earlier quoted context omitted.

Wow, for a company that boasts "no bullshit", only offering a month after destroying data and backups seems a little tone deaf Edit: in fairness, I'm not sure how exactly you would quantify such a loss anyway...

It sounds like they didn’t have any backups at all but rather relied on a active-active replication link to a secondary storage. Edit: who knows it may be related to the HPE issue. https://www.bleepingcomputer.com/news/hardware/hp-warns-that...

What baffles me is that there seems to be no way for either the customer or a data-recovery company to flash a new firmware onto the drive after it has failed. Someone there wanted to spare the few millicents of copper trace for a JTAG port?!

Re: Gandi loses data, customers told to use their own backups

#33
post #9

I understand people might be upset because they lost data, but as a sysadmin, my reaction is "ooh shit, poor guys, that must be a horrible week"... And honestly, if you don't keep data of stuff you host on a server provider like this, you kind of get what you deserve...

Shit does happen, but pretending like it's not a big deal and not providing a solid RCA seems to be what's really annoying about their reaction.

Re: Gandi loses data, customers told to use their own backups

#34
post #6

Gandi have something of a cult following, but in my only experience with them they literally lost my domain name during an inbound transfer. Their response was awful and rude and completely unprofessional. I never got my domain back. Based on that experience, this incident doesn’t surprise me at all.

I’ve always been a little confused about their cult following given their unfriendly terms — arbitrary domain cancellation based on adult material for example — which are fair terms to have if that’s their ethics but it seems at odds with the typical pro freedom expectations many people in technology hold.

Any details on this? All I found while searching for this was Gandi explicitly advertising gTLDs designed for adult content...

Do they have that in their terms? Independently of that, do they have a history of doing that?

Re: Gandi loses data, customers told to use their own backups

#35
post #14
post #9

I understand people might be upset because they lost data, but as a sysadmin, my reaction is "ooh shit, poor guys, that must be a horrible week"... And honestly, if you don't keep data of stuff you host on a server provider like this, you kind of get what you deserve...

No you don't. While agree everyone should have their own backups, you should expect your hosting company to properly replicate and backup their datacenters.

I don't, actually, expect them to do so. But even if I would, and Gandi, here, were doing backups and replications, no one is immune from errors and catastrophes.

Pretending that the cloud is permanent in infallible is extremely dangerous. I would seriously question the competence of any sysadmin relying on this as a base principle.

Sure, they screwed up, but this stuff happens. We should actually be happy it happens "only" on a "small-ish" provider like Gandi and not an entire AZ at Amazon.

Can't wait for that shoe to drop, I'll bring the popcorn, if there's anything left of civilization then...

Re: Gandi loses data, customers told to use their own backups

#36
post #14
post #9

I understand people might be upset because they lost data, but as a sysadmin, my reaction is "ooh shit, poor guys, that must be a horrible week"... And honestly, if you don't keep data of stuff you host on a server provider like this, you kind of get what you deserve...

No you don't. While agree everyone should have their own backups, you should expect your hosting company to properly replicate and backup their datacenters.

That is not the industry standard for web hosting. Never has been, never will be.

Backups aren't free. Replication isn't free. DR isn't free. If a customer isn't paying a premium for them, they aren't getting them. Read the terms of service.

Re: Gandi loses data, customers told to use their own backups

#37
post #9

I understand people might be upset because they lost data, but as a sysadmin, my reaction is "ooh shit, poor guys, that must be a horrible week"... And honestly, if you don't keep data of stuff you host on a server provider like this, you kind of get what you deserve...

> You get what you deserve Sure, let's blame the victims here; that's effective and helpful.

culpability isn't zero-sum, everyone can have some. some entities deserve a lot, others deserve just a teeny tiny little bit.

for purposes of keeping your data safe, your cloud provider is just one, single, copy of your data. all of their redundancies and backups and whatnot are for _their_ convenience, not yours, regardless of the marketing copy.

(they can decide to intentionally delete your data because they think you didn't pay. no amount of RAID and georedundant backups on their part will help you then.)

Re: Gandi loses data, customers told to use their own backups

#39
post #17

Dear customer, This mail is a follow-up to the previous email we sent (on January 8th, 2020) on this topic. As a reminder, yesterday, we experienced an incident on a storage unit at our LU-BI1 datacenter, located in Luxembourg. Despite the replication systems in place, and the combined efforts of our technical teams throughout the night, we were unable to reover the data that was lost on the impacted storage unit. We…

> This type of incident is extremely rare in the web hosting industry. Why would they include that sentence? Are they trying to imply it is rare for them because it is rare for the industry? Are they saying they are not as good as the industry, so customers should move to other providers? Or are they trying to show they apply the same inattention to their customer communication as they apply to their data backup/reco…

I'm not sure why you've been downvoted for this. I thought the same.

I read it as: "This type of incident is extremely rare in the web hosting industry, because apparently the overwhelming majority of our competitors aren't capable of fucking up as badly as we just did."

Doesn't inspire confidence at all, IMO.

Re: Gandi loses data, customers told to use their own backups

#40
post #6

Gandi have something of a cult following, but in my only experience with them they literally lost my domain name during an inbound transfer. Their response was awful and rude and completely unprofessional. I never got my domain back. Based on that experience, this incident doesn’t surprise me at all.

I’ve always been a little confused about their cult following given their unfriendly terms — arbitrary domain cancellation based on adult material for example — which are fair terms to have if that’s their ethics but it seems at odds with the typical pro freedom expectations many people in technology hold.

They put a rude word on their homepage, that makes them edgy and cool and anti-corporation!
Post reply on HN