Gandi loses data, customers told to use their own backups • Hacker News Board

Re: Gandi loses data, customers told to use their own backups

#341

Post by aspectmin » Fri, Jan 10, 2020, 1:44 AM UTC

I helped co-found a large Dropbox-like white label product. We used AWS and especially s3 for storage.

After many many years of experience with systems, I made sure we had as many possible ways to recover user data as we could. The initial solution was a large Postgres database for all the metadata/indices and s3 for the actual storage.

Despite much pushback we built in little things like an individual meta file on the file system for each file we stored. That way, if we lost the Postgres dB for any reason, we could create a script to rebuild the dB and restore access avoiding massive counts of orphaned files. A simple and probably stupid solution but...

Well guess what - the DB got corrupted and after some ado, we restored all access and none of our customers lost anything.

No it’s not full backups but...

Re: Gandi loses data, customers told to use their own backups

#342

Post by smnrchrds » Fri, Jan 10, 2020, 2:20 AM UTC

jiggawatts wrote: post #337

yjftsjthsd-h wrote: post #326

Earlier quoted context omitted.

> Ask yourself this: Could your organisation recover if some malicious admin simply deleted all Azure Resource Manager resources in one go using PowerShell? We have streaming replicas for hot data AND regular snapshots shipped to offsite cold storage, because RAID is not a backup . If we experienced an equivalent event, we'd be fine.

The equivalent scenario to recovering from a bulk erasure of all Azure RM resources is this: How long will it take you to recover if someone deleted your switch configs, reset the SAN to factory defaults, wiped you firewall rules, deleted you Active Directory accounts (or equivalent), and then ran a secure erase on every every physical server just to raze everything to the ground and salt the earth? I mean in wall-cl…

> The equivalent scenario to recovering from a bulk erasure of all Azure RM resources is this

It just occurred to me how much easier it is to wipe everything in the cloud age than the on-prem age. Doing all the things you said for on-prem takes some serious effort. Some, like factory resets, may be impossible without individual physical access. You would probably be discovered and stopped before you can inflict much damage. In the cloud age however, it takes orders of magnitude less time and effort to inflict the same damage.

It is kinda like how much easier it is to steal data now. Before the digital age, stealing as much data as Equifax hack would have required moving truckloads of paper without being discovered. It was simply impossible to pull it off in reality. In the digital age, however, we have accepted massive data leaks as not only possible, but unavoidable.

Re: Gandi loses data, customers told to use their own backups

#343

Post by Operyl » Fri, Jan 10, 2020, 2:23 AM UTC

smnrchrds wrote: post #333

capableweb wrote: post #275

Earlier quoted context omitted.

Sounds more like a bug than anything. Why would they want to not make it easier for you if they can? Seems you missed my point though. Both of our anecdotes doesn't really say anything, in terms of if Gandi is good or bad.

His anecdote does say something though. It suggests that Gandi has a "if we have a bug, it's your problem not ours, sucks to be you" policy, which is exactly what has happened with this data loss issue as well. Actions speak louder than words. Google famously has a "we don't have bugs, you just don't know how to use it, talk to the hand" policy for example. It is better to learn about the policies due to minor issues…

I’m a gal, but exactly. I really wanted to support a company at the time who was supporting the community (they were a freenode sponsor), but I just hated dealing with the stress and potential that my domains could just disappear over night.

Re: Gandi loses data, customers told to use their own backups

#344

Post by [deleted] » Fri, Jan 10, 2020, 2:34 AM UTC

Operyl wrote: post #343

smnrchrds wrote: post #333

Earlier quoted context omitted.

His anecdote does say something though. It suggests that Gandi has a "if we have a bug, it's your problem not ours, sucks to be you" policy, which is exactly what has happened with this data loss issue as well. Actions speak louder than words. Google famously has a "we don't have bugs, you just don't know how to use it, talk to the hand" policy for example. It is better to learn about the policies due to minor issues…

I’m a gal, but exactly. I really wanted to support a company at the time who was supporting the community (they were a freenode sponsor), but I just hated dealing with the stress and potential that my domains could just disappear over night.

[deleted]

Re: Gandi loses data, customers told to use their own backups

#345

Post by technion » Fri, Jan 10, 2020, 2:50 AM UTC

dkarras wrote: post #251

This is like living in an alternate universe, I've been heavily involved in all things programming and webdev for years, following trends and whatnot and it is literally the first time I'm hearing of this particular company. What is (was?) so special about them that they attracted the HN crowd can someone briefly explain? Why would I buy domain from them when something like namecheap, even google domains exists? Why…

I wrote out a list of things I needed in a domain registrar and once you include U2F logons and DNSSEC support, you find yourself in a very limited space.

Re: Gandi loses data, customers told to use their own backups

#346

Post by yjftsjthsd-h » Fri, Jan 10, 2020, 2:53 AM UTC

jiggawatts wrote: post #337

yjftsjthsd-h wrote: post #326

Earlier quoted context omitted.

> Ask yourself this: Could your organisation recover if some malicious admin simply deleted all Azure Resource Manager resources in one go using PowerShell? We have streaming replicas for hot data AND regular snapshots shipped to offsite cold storage, because RAID is not a backup . If we experienced an equivalent event, we'd be fine.

The equivalent scenario to recovering from a bulk erasure of all Azure RM resources is this: How long will it take you to recover if someone deleted your switch configs, reset the SAN to factory defaults, wiped you firewall rules, deleted you Active Directory accounts (or equivalent), and then ran a secure erase on every every physical server just to raze everything to the ground and salt the earth? I mean in wall-cl…

I think you're moving the goalposts. Gandi didn't lose all their servers and all the networking hardware and all the storage. They lost what sounds like a single replicated volume. If, y'know, all of their datacenters burned down at once, or an attacker got access and deleted their PaaS account, I think we'd all be a lot more sympathetic

Re: Gandi loses data, customers told to use their own backups

#347

Post by glenneroo » Fri, Jan 10, 2020, 3:23 AM UTC

jiggawatts wrote: post #336

bscphil wrote: post #324

Earlier quoted context omitted.

Everything you say here is true, but at the same time it's just a fact that Gandi lost a lot of customers' data, and AWS, GCP, and Azure have never (as far as I know) lost a significant amount of it at once. You can talk about theoretical responsibility for data, and it's true, you are responsible for having backups of your data, no matter how many "9s" the service has, but the basic fact is that some services have b…

Back in the early days GMail lost customer data due to storage corruption. It has happened. The rarity is immaterial, the responsibility for data protection lies with you, not them.

Well to be fair Gmail was still in Beta ;)

Re: Gandi loses data, customers told to use their own backups

#348

Post by rorykoehler » Fri, Jan 10, 2020, 3:33 AM UTC

user1980 wrote: post #266

rorykoehler wrote: post #56

Earlier quoted context omitted.

Same boat. As much as I hate to give Jeff Bezos another penny I can't look further than AWS for everything at this stage.

Don't worry, if you purchase a domain with a .biz (or 300 other junk-tier tLDs) extension from Amazon Route 53, Gandi still gets paid[1]. [1] https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/re...

Huh. I wonder how they will react to this. Thanks for highlighting that.

Re: Gandi loses data, customers told to use their own backups

#349

Post by dragonwriter » Fri, Jan 10, 2020, 3:39 AM UTC

smnrchrds wrote: post #342

jiggawatts wrote: post #337

Earlier quoted context omitted.

The equivalent scenario to recovering from a bulk erasure of all Azure RM resources is this: How long will it take you to recover if someone deleted your switch configs, reset the SAN to factory defaults, wiped you firewall rules, deleted you Active Directory accounts (or equivalent), and then ran a secure erase on every every physical server just to raze everything to the ground and salt the earth? I mean in wall-cl…

> The equivalent scenario to recovering from a bulk erasure of all Azure RM resources is this It just occurred to me how much easier it is to wipe everything in the cloud age than the on-prem age. Doing all the things you said for on-prem takes some serious effort. Some, like factory resets, may be impossible without individual physical access. You would probably be discovered and stopped before you can inflict much…

> It just occurred to me how much easier it is to wipe everything in the cloud age than the on-prem age.

It's easier for physical facility damage to a single facility (whether hostile action or natural disaster) to wipe everything out in an on-prem setup than in the cloud, where multi-DC redundancy is a click away. But, sure, it's easier to wipe out data without physically destroying equipment in the cloud.

Re: Gandi loses data, customers told to use their own backups

#350

Post by austhrow743 » Fri, Jan 10, 2020, 3:45 AM UTC

dylz wrote: post #213

dcchambers wrote: post #85

Earlier quoted context omitted.

Wow. I've never used Gandi but I have seen it recommended before as a low-cost option. I will actively encourage people to avoid it from now on. That's scary.

Gandi has never been a low cost option, they've always been on the high to extreme higher end of things for individual cost... Especially for random ccTLDs, they're often significantly more expensive than the alternatives. Random selections for domains: .ru is $1-3 most anywhere else, Gandi is $18.

High to extreme higher end would be something like MarkMonitor.

Gandi might be better than some of the other low touch, self service domain providers but its definitely still in the same ballpark. $18/year still means they're losing money if they ever need to pick up the phone for you. It's not a price point that works with "higher end".