Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gandi loses data, customers told to use their own backups

status.gandi.net

Post reply on HN
Print view View original

81–90 of 389 posts

Re: Gandi loses data, customers told to use their own backups

#81

Post by terom »

terom wrote: post #54

From the incident timeline: > we have a problem to import zfs pool on the unit storage I really want to know what went wrong to a) break ZFS b) prevent recovery from backup.

Re myself, it looks like they're using FreeBSD-based ZFS filers with iSCSI/NFS exports using a user-spce NFS server:

* https://news.gandi.net/en/2019/09/exporters-detect-micro-inc...

> Gandi’s storage infrastructure consists of two environments: one for IaaS and one for PaaS. Both are based on FreeBSD-based storage units (filers), that stock each volume (disk) as though it were a ZFS volume.

* https://news.gandi.net/en/2019/03/tracking-a-storage-issue-l...

* https://www.bsdcan.org/2016/schedule/attachments/351_FreeBSD...

No mention of what they're doing for backups / "replication systems", unsurprisingly/unfortunately. I'm anxious to know what the failure mode for `zfs send | zfs receive` replication is here?

Re: Gandi loses data, customers told to use their own backups

#82

Post by veeti »

celticmusic wrote: post #79
aosaigh wrote: post #13

I don't have hosting with Gandi, but I do use them for domains and DNS. I'll be considering migrating my domains from them after this. Their response to this is exceptionally poor. To say essentially "this could happen to any other web host" it nonsense. I've never had this happen with any of the providers I've used for hosting and I'd be very angry if I had just lost an entire VPS. The fact that they've lost all sna…

I had an incident similar to this with linode, which is why I use and recommend Digital Ocean nowadays. My machine going away because you had hardware issues isn't my problem, and I'll spend my money on a more competent company.

I had the exact same experience on Digital Ocean. Attempted to resize a VPS, the process got stuck for eternity, and support tells me all data is lost.

Always have your own offsite backups.

Re: Gandi loses data, customers told to use their own backups

#83

Post by oefrha »

arpa wrote: post #48
oefrha wrote: post #31

Earlier quoted context omitted.

Often times the backup provider is the hosting provider, whom you have to trust. (This extends all the way from big clouds like AWS and GCE to small providers like Linode and DO). Having an external backup can be unreasonably expensive due to ridiculous egress costs.

You can still back up to the same providers' different data center. Two data centers failing simultaneously is very unlikely.

Not always an option. For instance, I use Linode’s backup service and it can only back up to the same data center (although it is said to live on a separate system).

Re: Gandi loses data, customers told to use their own backups

#84

Post by y4mi »

Eikon wrote: post #64
y4mi wrote: post #61

Earlier quoted context omitted.

You might want to read the Wikipedia definition, because you're technically mistaken. https://en.m.wikipedia.org/wiki/Backup

I'm not, you can't do backup without replicating data, hence, if you do backup, you are doing replication.

It can be, it just doesn't have to be

Let's say a typical admin of a small shop wants to backup his postgres database.

The first thing he'll use is probably pg_dumpall which he'll output to a storage.

No replication involved. The backup is just a bunch of sql statements to recover the last known state of the database. It's a different kind of format however, which -by definition- isn't a replica anymore.

(And this process has several caveat's-one of which is that it can produce unusable dumps in some rare cases and isn't complete. users, triggers etc aren't dumped iirc.. could be wrong there)

Re: Gandi loses data, customers told to use their own backups

#85

Post by dcchambers »

EnderMB wrote: post #28

Oof, this Twitter thread looks particularly bad, especially the response from the official Gandi account. https://twitter.com/andreaganduglia/status/12151991477012316... While I appreciate that there are real people behind these companies that are probably having a really rough time right now, the criticism that Gandi are getting as a company is justified - and if Gandi are truly a "no bullshit" company they need to…

Wow. I've never used Gandi but I have seen it recommended before as a low-cost option. I will actively encourage people to avoid it from now on. That's scary.

Re: Gandi loses data, customers told to use their own backups

#86

Post by LaGrange »

tbyehl wrote: post #69
rahuldottech wrote: post #51

Earlier quoted context omitted.

In this case, the customer did pay for it: https://twitter.com/andreaganduglia/status/12152083871699804... See full thread. Snapshots are marketed as backups.

So? Intelligent people can argue all day about whether a snapshot should be considered a backup or not, but it won't change the fact that a snapshot doesn't provide any protection from a failure in the underlying storage and it's ridiculously foolish for the owner of data to solely rely on snapshots as their backup strategy.

They literally use the word "backup." I wouldn't _normally_ expect snapshots to function as backups, but once they market them as such, I do. Yeah, sure, it's probably yet another case of a sales team getting over eager and taking over the company, but that's why if you value your ethics _at all_ you keep tabs on WTF the sales are doing.

Re: Gandi loses data, customers told to use their own backups

#87

Post by moralestapia »

Whoops, so long with the "no bullshit" policy.

I stopped using them a while ago but for a different reason. I used to use their website to check availability/whois for domains that I was interested in buying. If it was available I didn't buy it at the time but until I finished the website/app whatever I was going to put there, this took me a few months obviously. It happened to me that when I was finally ready the domain had already been sold to someone else. This repeated five times during six or so years. Now, I know, "someone else could have thought the same thing" but I find it very hard to believe that it happens so often. These domains were a bit of niche words that were not hot topics at the time, some of them using fairly uncommon TLDs (like .one). Another weird thing is that they were always registered to someone living/or doing business at India, and it was a fairly simple landing page with a "contact me" link. I'm a bit superstitious so I don't think it was a coincidence.

Now, I don't think this is a GANDI problem per se, but my theory is that they share this information (who is looking for which domains) with marketers or something like that, or maybe it was a rogue employee trying to make some money squatting domains. I would have expected this from BigDaddy or similar sharks, but from a company whose motto is "no bullshit" I had much better hope. Anyway, I decided to move (to namecheap if you're wondering) and surprisingly the problem went away.

Re: Gandi loses data, customers told to use their own backups

#89

Post by alpaca128 »

anarcat wrote: post #35
aosaigh wrote: post #14

Earlier quoted context omitted.

No you don't. While agree everyone should have their own backups, you should expect your hosting company to properly replicate and backup their datacenters.

I don't, actually, expect them to do so. But even if I would, and Gandi, here, were doing backups and replications, no one is immune from errors and catastrophes. Pretending that the cloud is permanent in infallible is extremely dangerous. I would seriously question the competence of any sysadmin relying on this as a base principle. Sure, they screwed up, but this stuff happens. We should actually be happy it happens…

> Gandi, here, were doing backups and replications

As far as I understand correctly they only made snapshots on the same machine, which is why there's trouble to begin with.

Considering they're currently "reminding" customers that backups are an industry standard right after losing data due to missing backups I wouldn't just shrug it off.

Re: Gandi loses data, customers told to use their own backups

#90

Post by code4tee »

danr4 wrote: post #49
EnderMB wrote: post #28

Oof, this Twitter thread looks particularly bad, especially the response from the official Gandi account. https://twitter.com/andreaganduglia/status/12151991477012316... While I appreciate that there are real people behind these companies that are probably having a really rough time right now, the criticism that Gandi are getting as a company is justified - and if Gandi are truly a "no bullshit" company they need to…

Screenshotted in case (when) they delete it https://i.imgur.com/s3R1VVc.png Using memes after permanently losing customer data is extremely disrespectful.

Why are they @&$ing around on Twitter when they should be fixing the damn problem. Unbelievable.
Post reply on HN