Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gandi loses data, customers told to use their own backups

status.gandi.net

Post reply on HN
Print view View original

11–20 of 389 posts

Re: Gandi loses data, customers told to use their own backups

#11

Post by Shalle135 »

petee wrote: post #3
webrobots wrote: post #2

Dear customer, This mail is a follow-up to the previous email we sent (on January 8th, 2020) on this topic. As a reminder, yesterday, we experienced an incident on a storage unit at our LU-BI1 datacenter, located in Luxembourg. Despite the replication systems in place, and the combined efforts of our technical teams throughout the night, we were unable to reover the data that was lost on the impacted storage unit. We…

Wow, for a company that boasts "no bullshit", only offering a month after destroying data and backups seems a little tone deaf Edit: in fairness, I'm not sure how exactly you would quantify such a loss anyway...

It sounds like they didn’t have any backups at all but rather relied on a active-active replication link to a secondary storage.

Edit: who knows it may be related to the HPE issue.

https://www.bleepingcomputer.com/news/hardware/hp-warns-that...

Re: Gandi loses data, customers told to use their own backups

#12

Post by quaquaqua1 »

djmobley wrote: post #6

Gandi have something of a cult following, but in my only experience with them they literally lost my domain name during an inbound transfer. Their response was awful and rude and completely unprofessional. I never got my domain back. Based on that experience, this incident doesn’t surprise me at all.

In the year 2020, it's becoming increasingly impossible to trust anyone to do nearly anything (in my opinion of course).

The courts are too expensive. The culture of taking pride in one's work maybe is disappearing.

For the most crucial parts of doing business/living life, we are required to trust someone else. For example, I can't just go and make my own cell phone tower or ICANN.

And yet I can't even trust those entities to get it right.

Re: Gandi loses data, customers told to use their own backups

#13

Post by aosaigh »

I don't have hosting with Gandi, but I do use them for domains and DNS. I'll be considering migrating my domains from them after this.

Their response to this is exceptionally poor. To say essentially "this could happen to any other web host" it nonsense. I've never had this happen with any of the providers I've used for hosting and I'd be very angry if I had just lost an entire VPS. The fact that they've lost all snapshots as well (which are advertised as backups of the underlying volume) is unforgiveable.

Re: Gandi loses data, customers told to use their own backups

#14

Post by aosaigh »

anarcat wrote: post #9

I understand people might be upset because they lost data, but as a sysadmin, my reaction is "ooh shit, poor guys, that must be a horrible week"... And honestly, if you don't keep data of stuff you host on a server provider like this, you kind of get what you deserve...

No you don't. While agree everyone should have their own backups, you should expect your hosting company to properly replicate and backup their datacenters.

Re: Gandi loses data, customers told to use their own backups

#15

Post by mosselman »

anarcat wrote: post #9

I understand people might be upset because they lost data, but as a sysadmin, my reaction is "ooh shit, poor guys, that must be a horrible week"... And honestly, if you don't keep data of stuff you host on a server provider like this, you kind of get what you deserve...

The sysadmins over there probably have a whole list of stuff that should actually have been done, but management never gave them time to do. Then this happened and they were proven right. Their reward? Working a lot of overtime probably.

Re: Gandi loses data, customers told to use their own backups

#16

Post by jnwatson »

Shalle135 wrote: post #11
petee wrote: post #3

Earlier quoted context omitted.

Wow, for a company that boasts "no bullshit", only offering a month after destroying data and backups seems a little tone deaf Edit: in fairness, I'm not sure how exactly you would quantify such a loss anyway...

It sounds like they didn’t have any backups at all but rather relied on a active-active replication link to a secondary storage. Edit: who knows it may be related to the HPE issue. https://www.bleepingcomputer.com/news/hardware/hp-warns-that...

In other words, RAID is not backup.

Re: Gandi loses data, customers told to use their own backups

#17

Post by abtinf »

webrobots wrote: post #2

Dear customer, This mail is a follow-up to the previous email we sent (on January 8th, 2020) on this topic. As a reminder, yesterday, we experienced an incident on a storage unit at our LU-BI1 datacenter, located in Luxembourg. Despite the replication systems in place, and the combined efforts of our technical teams throughout the night, we were unable to reover the data that was lost on the impacted storage unit. We…

> This type of incident is extremely rare in the web hosting industry.

Why would they include that sentence? Are they trying to imply it is rare for them because it is rare for the industry? Are they saying they are not as good as the industry, so customers should move to other providers? Or are they trying to show they apply the same inattention to their customer communication as they apply to their data backup/recovery practices?

This kind of data loss should simply never happen. It’s one thing to say “it will take us up to 30 days to restore your data because our fast recovery options aren’t working and we have to bring up cold archives”, it’s entirely another to say “your data is gone, tough”.

Re: Gandi loses data, customers told to use their own backups

#18

Post by briffle »

Eikon wrote: post #10

> We sincerely apologize for the inconvenience that this situation has caused. This type of incident is extremely rare in the web hosting industry. Why are they speaking of the "industry" as a whole when they are to blame? It's even crazier they are not even explaining the source of the data loss and why the "replication systems" didn't help. IHMO they are trying to sweep this event under the carpet. They should inst…

In the last few years, I have seen many people confuse replication with backups. People see them as the same thing, but they really aren't. Even with snapshots, if the devices are the same, they might have the same firmware bug, etc.

Re: Gandi loses data, customers told to use their own backups

#19

Post by scohesc »

atlasAUPrivacy wrote: post #5
mmoez wrote: post #4

Earlier quoted context omitted.

A promo code in exchange of your data loss. What a bargain!

“Please keep trusting us to host your data”

You really shouldn't trust anyone hosting your data. Always have backups!

Re: Gandi loses data, customers told to use their own backups

#20

Post by pablasso »

anarcat wrote: post #9

I understand people might be upset because they lost data, but as a sysadmin, my reaction is "ooh shit, poor guys, that must be a horrible week"... And honestly, if you don't keep data of stuff you host on a server provider like this, you kind of get what you deserve...

Even if it is a bad practice not to have your own backups, no one is at fault here but Gandi
Post reply on HN