Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gandi loses data, customers told to use their own backups

status.gandi.net

Post reply on HN
Print view View original

141–150 of 389 posts

Re: Gandi loses data, customers told to use their own backups

#141

Post by specialist »

quaquaqua1 wrote: post #12
djmobley wrote: post #6

Gandi have something of a cult following, but in my only experience with them they literally lost my domain name during an inbound transfer. Their response was awful and rude and completely unprofessional. I never got my domain back. Based on that experience, this incident doesn’t surprise me at all.

In the year 2020, it's becoming increasingly impossible to trust anyone to do nearly anything (in my opinion of course). The courts are too expensive. The culture of taking pride in one's work maybe is disappearing. For the most crucial parts of doing business/living life, we are required to trust someone else. For example, I can't just go and make my own cell phone tower or ICANN. And yet I can't even trust those en…

Decreasing trust increases transaction costs.

There's got to be a measurable (negative) economic impact.

Re: Gandi loses data, customers told to use their own backups

#142

Post by gdxhyrd »

rjmunro wrote: post #80
CydeWeys wrote: post #57

Earlier quoted context omitted.

Fortunately, git is a DVCS, so anyone who checks out a repo has a complete copy of it. Now, granted, it'd be a huge pain to track down all the people who had copies of the 1,500 different repos, and try to find as up-to-date as possible of a version of each, but I doubt they got anywhere close to potentially losing all their source code. Incidentally this shows why it's a good idea to sync your repo to GitHub, even i…

> 1,500 separate repositories?! That sounds way overkill. I wonder if they'd benefit from having a monorepo. No it doesn't. Github has at least 20 million public repositories. Would they benefit by combining them into a monorepo?

GP is talking about the KDE project, not the entirety of GitHub.

And yes, a monorepo is usually the best approach in most cases for a project or even an entire company.

Re: Gandi loses data, customers told to use their own backups

#143

Post by notyourday »

alpaca128 wrote: post #89
anarcat wrote: post #35

Earlier quoted context omitted.

I don't, actually, expect them to do so. But even if I would, and Gandi, here, were doing backups and replications, no one is immune from errors and catastrophes. Pretending that the cloud is permanent in infallible is extremely dangerous. I would seriously question the competence of any sysadmin relying on this as a base principle. Sure, they screwed up, but this stuff happens. We should actually be happy it happens…

> Gandi, here, were doing backups and replications As far as I understand correctly they only made snapshots on the same machine, which is why there's trouble to begin with. Considering they're currently "reminding" customers that backups are an industry standard right after losing data due to missing backups I wouldn't just shrug it off.

That's probably because they bought into the sales pitches of the likes of EMC. It's a nice pitch and in most of the cases it works exactly like EMC promises. Snapshots work great, data is always recovered, etc, etc, etc.

The fun, of course, starts that one time when it does not work and you realize that no one looked at the corner case that bit you.

Re: Gandi loses data, customers told to use their own backups

#144

Post by itake »

teh_klev wrote: post #130

The key question is, did Gandi offer and explicit backup service for your data on their plans? I just had a look and I don't see this being offered. As a former hosting engineer, at the risk of pissing on everyone's outrage parade, but unless an explicit guarantee of a backup is included in your plan's contract, or you can pay for backups as a bolt-on, then if you've lost data it's your fault for not planning for thi…

> Easily recover backups of previous versions of your website's files, thanks to our automatic Snapshots system. It's free!

https://docs.gandi.net/en/simple_hosting/common_operations/s...

They are supposed to be providing backups.

Re: Gandi loses data, customers told to use their own backups

#145

Post by tomatocracy »

terom wrote: post #81
terom wrote: post #54

From the incident timeline: > we have a problem to import zfs pool on the unit storage I really want to know what went wrong to a) break ZFS b) prevent recovery from backup.

Re myself, it looks like they're using FreeBSD-based ZFS filers with iSCSI/NFS exports using a user-spce NFS server: * https://news.gandi.net/en/2019/09/exporters-detect-micro-inc... > Gandi’s storage infrastructure consists of two environments: one for IaaS and one for PaaS. Both are based on FreeBSD-based storage units (filers), that stock each volume (disk) as though it were a ZFS volume. * https://news.gandi.net/…

Sounds very much like they weren't doing zfs send | zfs receive to anything sufficiently physically separated. For example, if you send and receive in the same pool, it's replication but still leaves you vulnerable to issues where the pool can't be imported due to corruption in the wrong places (it can happen) or significant hardware failure (eg a PSU fault that takes out too many of the drives in the pool).

Re: Gandi loses data, customers told to use their own backups

#146

Post by jolmg »

oefrha wrote: post #83
arpa wrote: post #48

Earlier quoted context omitted.

You can still back up to the same providers' different data center. Two data centers failing simultaneously is very unlikely.

Not always an option. For instance, I use Linode’s backup service and it can only back up to the same data center (although it is said to live on a separate system).

They could mean using regular data transfer (i.e. using something like rsync instead of the provider's backup service). Maybe egress costs among servers from the same provider are reduced or nullified.

From[1]:

> Traffic over the private network does not count against your monthly quota.

I wonder how private addresses are setup by Linode.

[1] https://www.linode.com/docs/platform/billing-and-support/net...

Re: Gandi loses data, customers told to use their own backups

#147

Post by anarazel »

y4mi wrote: post #84
Eikon wrote: post #64

Earlier quoted context omitted.

I'm not, you can't do backup without replicating data, hence, if you do backup, you are doing replication.

It can be, it just doesn't have to be Let's say a typical admin of a small shop wants to backup his postgres database. The first thing he'll use is probably pg_dumpall which he'll output to a storage. No replication involved. The backup is just a bunch of sql statements to recover the last known state of the database. It's a different kind of format however, which -by definition- isn't a replica anymore. (And this pr…

> (And this process has several caveat's-one of which is that it can produce unusable dumps in some rare cases and isn't complete. users, triggers etc aren't dumped iirc.. could be wrong there)

Triggers are dumped, users need pg_dumpall (as they live across multiple databases, same with tablespaces).

Re: Gandi loses data, customers told to use their own backups

#148

Post by newscracker »

This seems like data has been lost from servers hosting sites/services.

Since Gandi is mostly known for domain registrations and DNS, I'm curious if you (as an individual who hosts websites/online services somewhere on the web) backup your site's DNS records periodically (or whenever they're changed). What if your authoritative name server lost data and all the caches of those records across geographies expire while you're asleep/away? If you do back these up regularly, how do you do it in an automated way on a *nix system? I found this article [1] when I searched about this, but it's not a simple shell script. The scripts that I did find on some of the Stackexchange sites seemed to have specific subdomain names hardcoded.

[1]: http://www.programblings.com/2012/07/23/do-you-back-up-your-...

Re: Gandi loses data, customers told to use their own backups

#149

Post by martius »

abtinf wrote: post #17
webrobots wrote: post #2

Dear customer, This mail is a follow-up to the previous email we sent (on January 8th, 2020) on this topic. As a reminder, yesterday, we experienced an incident on a storage unit at our LU-BI1 datacenter, located in Luxembourg. Despite the replication systems in place, and the combined efforts of our technical teams throughout the night, we were unable to reover the data that was lost on the impacted storage unit. We…

> This type of incident is extremely rare in the web hosting industry. Why would they include that sentence? Are they trying to imply it is rare for them because it is rare for the industry? Are they saying they are not as good as the industry, so customers should move to other providers? Or are they trying to show they apply the same inattention to their customer communication as they apply to their data backup/reco…

I think it was meant to say "nobody is infallible", these events are extremely rare, but they /will/ occur, even if you're a customer of the best and biggest players.

Re: Gandi loses data, customers told to use their own backups

#150

Post by petee »

danr4 wrote: post #49
EnderMB wrote: post #28

Oof, this Twitter thread looks particularly bad, especially the response from the official Gandi account. https://twitter.com/andreaganduglia/status/12151991477012316... While I appreciate that there are real people behind these companies that are probably having a really rough time right now, the criticism that Gandi are getting as a company is justified - and if Gandi are truly a "no bullshit" company they need to…

Screenshotted in case (when) they delete it https://i.imgur.com/s3R1VVc.png Using memes after permanently losing customer data is extremely disrespectful.

"Julie Pelloille @juliepelloille Replying to @gandi_net @andreaganduglia and 4 others

This post was disrespectful. It's not an excuse, but this is a stressful situation and the thread was getting heated. Either way, I truly regret posting it and it was my decision alone to do so. Please don't take this as representative of the high standard Gandi sets"

"That said, for the sake of transparency, we won't be deleting the tweet -- Julie"

Post reply on HN