Gandi loses data, customers told to use their own backups • Hacker News Board

Re: Gandi loses data, customers told to use their own backups

#331

Post by unishark » Fri, Jan 10, 2020, 12:45 AM UTC

mystcb wrote: post #257

brianwawok wrote: post #214

Earlier quoted context omitted.

I suspect a human gets a dump of them and decides which to pay the $10 for. For example, asdasdahbdajsdbajdbhsbdahsdd.com... not worth the $10 ireallylikechicken.com... maybe worth the $10? (ireallylikechicken.com is available, go squat it and get rich)

I am sure, in the past, one of the domain registrars took the liberty of actually registering your searched domain, deliberately, so that you had to go through them to get the domain later on? - I can't remember who it was, but it was an automated process. I know it was very shortly stopped once people complained, but it goes to show that it has been done before. Getting rich off domain's - sounds like a solid busine…

I remember something about registrars doing this so the domain you wanted wouldn't get snatched by someone else while you were in the process of buying it. Made more sense back in the time when far more basic names were available and such collisions were probably common for high-value domain names. Now it's two people finding the same needle in a haystack.

Re: Gandi loses data, customers told to use their own backups

#332

Post by unishark » Fri, Jan 10, 2020, 12:54 AM UTC

zulgan wrote: post #219

I had a co-worker who was super chill during outages; especially at night, we were 10-15 people on the call fixing issues related to his work almost monthly. those outages costed millions of euros, and he never picked up his phone at night, once I asked him why he never picks up, he told me: "I used to be a general surgeon, when someone calls me people die. Relax, nobody is dying during our outages." now I think I am…

"I used to be a general surgeon, when someone calls me people die."

Hence he's not a surgeon anymore.

Re: Gandi loses data, customers told to use their own backups

#333

Post by smnrchrds » Fri, Jan 10, 2020, 1:01 AM UTC

capableweb wrote: post #275

Operyl wrote: post #244

Earlier quoted context omitted.

It was a matter of them refusing to keep my identity on file, and the threatening tone of each ticket. It grew tiresome quickly.

Sounds more like a bug than anything. Why would they want to not make it easier for you if they can? Seems you missed my point though. Both of our anecdotes doesn't really say anything, in terms of if Gandi is good or bad.

His anecdote does say something though. It suggests that Gandi has a "if we have a bug, it's your problem not ours, sucks to be you" policy, which is exactly what has happened with this data loss issue as well.

Actions speak louder than words. Google famously has a "we don't have bugs, you just don't know how to use it, talk to the hand" policy for example. It is better to learn about the policies due to minor issues rather than major. OP learned of it early on and moved away with little trouble. Others did not learn until now and stayed, and now they are SOL.

Re: Gandi loses data, customers told to use their own backups

#334

Post by bigiain » Fri, Jan 10, 2020, 1:05 AM UTC

yjftsjthsd-h wrote: post #326

jiggawatts wrote: post #310

Just to play devil's advocate: This is in no way different to how Azure, AWS, and GCP operate. They don't have backups either. They too rely on n-way replication, a bit like a distributed RAID. All cloud providers make it absolutely clear, in black & white, that protection of your data is your responsibility , not theirs. What I find hilarious is that most cloud providers only provide built-in backup functionality fo…

> Ask yourself this: Could your organisation recover if some malicious admin simply deleted all Azure Resource Manager resources in one go using PowerShell? We have streaming replicas for hot data AND regular snapshots shipped to offsite cold storage, because RAID is not a backup . If we experienced an equivalent event, we'd be fine.

"We have 'Data gone? Sucks to be you!' as translated by our VC's lawyer buried in our T&Cs" -- most "disruptive startups", probably...

Re: Gandi loses data, customers told to use their own backups

#335

Post by bigiain » Fri, Jan 10, 2020, 1:09 AM UTC

Scoundreller wrote: post #282

swarnie_ wrote: post #238

Earlier quoted context omitted.

ireallylikechicken.com owner. I'll give you 20 for it?

I’ll give $1 for a 0.01% share.

I'll buy an option to take a 0.01% share in 12 months for $0.50

Re: Gandi loses data, customers told to use their own backups

#336

Post by jiggawatts » Fri, Jan 10, 2020, 1:13 AM UTC

bscphil wrote: post #324

jiggawatts wrote: post #310

Just to play devil's advocate: This is in no way different to how Azure, AWS, and GCP operate. They don't have backups either. They too rely on n-way replication, a bit like a distributed RAID. All cloud providers make it absolutely clear, in black & white, that protection of your data is your responsibility , not theirs. What I find hilarious is that most cloud providers only provide built-in backup functionality fo…

Everything you say here is true, but at the same time it's just a fact that Gandi lost a lot of customers' data, and AWS, GCP, and Azure have never (as far as I know) lost a significant amount of it at once. You can talk about theoretical responsibility for data, and it's true, you are responsible for having backups of your data, no matter how many "9s" the service has, but the basic fact is that some services have b…

Back in the early days GMail lost customer data due to storage corruption. It has happened.

The rarity is immaterial, the responsibility for data protection lies with you, not them.

Re: Gandi loses data, customers told to use their own backups

#337

Post by jiggawatts » Fri, Jan 10, 2020, 1:19 AM UTC

yjftsjthsd-h wrote: post #326

jiggawatts wrote: post #310

Just to play devil's advocate: This is in no way different to how Azure, AWS, and GCP operate. They don't have backups either. They too rely on n-way replication, a bit like a distributed RAID. All cloud providers make it absolutely clear, in black & white, that protection of your data is your responsibility , not theirs. What I find hilarious is that most cloud providers only provide built-in backup functionality fo…

> Ask yourself this: Could your organisation recover if some malicious admin simply deleted all Azure Resource Manager resources in one go using PowerShell? We have streaming replicas for hot data AND regular snapshots shipped to offsite cold storage, because RAID is not a backup . If we experienced an equivalent event, we'd be fine.

The equivalent scenario to recovering from a bulk erasure of all Azure RM resources is this:

How long will it take you to recover if someone deleted your switch configs, reset the SAN to factory defaults, wiped you firewall rules, deleted you Active Directory accounts (or equivalent), and then ran a secure erase on every every physical server just to raze everything to the ground and salt the earth?

I mean in wall-clock time, how long would it take your team to even figure out what is going on? Where would you start?

Would you recover the switch first, or the server that you use to authenticate to it using RADIUS or LDAP?

How will you securely connect to servers if your CRL and OCSP servers are down?

How will you get access to your passwords if your file server where the key blob is stored is saying "Insert boot disk"?

People think that disaster recovery is for "I deleted a folder".

Disaster recovery is for disasters.

Removing all Azure resources wipes everything. Your vNets... Poof! Your public IPs... Poof! Your internet-facing DNS zone... Poof! Your authentication credentials... Poof! Gone, gone, gone.

How do you plan to restore dynamic IP addresses to their original values?

How do you plan to restore DNS Zones that get assigned to 1 of 10 randomly selected server pools and hence have a 90% chance of requiring a change to the NS server glue records on restore?

Do you even know which order things would have to be restored in to prevent failures during a restore?

Could you possibly work out what is missing if you log on to your cloud portal and see the "Welcome to Azure, to get started click here" splash page?

Get it?

Re: Gandi loses data, customers told to use their own backups

#338

Post by LeonB » Fri, Jan 10, 2020, 1:19 AM UTC

moralestapia wrote: post #87

Whoops, so long with the "no bullshit" policy. I stopped using them a while ago but for a different reason. I used to use their website to check availability/whois for domains that I was interested in buying. If it was available I didn't buy it at the time but until I finished the website/app whatever I was going to put there, this took me a few months obviously. It happened to me that when I was finally ready the do…

Reports of reputable registrars front-running are persistent, but unfounded. Anytime I’ve looked into it, I’ve never seen any evidence for it.

If proven it would be a major blow to their business, so why would they try to snatch pennies from in front of a steam roller?

So I call b.s. on any reports of “the registrar noticed me searching for a domain and registered it”.

Re: Gandi loses data, customers told to use their own backups

#339

Post by jiggawatts » Fri, Jan 10, 2020, 1:24 AM UTC

teddyuk wrote: post #323

jiggawatts wrote: post #310

Just to play devil's advocate: This is in no way different to how Azure, AWS, and GCP operate. They don't have backups either. They too rely on n-way replication, a bit like a distributed RAID. All cloud providers make it absolutely clear, in black & white, that protection of your data is your responsibility , not theirs. What I find hilarious is that most cloud providers only provide built-in backup functionality fo…

Maybe not even malicious, maybe they just put in the wrong subscription ID :(

Yup.

This thought occurred to me when I was testing a bulk resource creation script.

My workflow in my lab tenant was:

1) Bulk create hundreds of resources 2) Bulk wipe everything 3) Go to step #1

Turned out, I had some objects with globally unique names that were now conflicting in the production tenant, so I had to wipe my lab.

I had already logged on to the production tenant, and I was so "trigger happy" that I very nearly ran my bulk-erase script against the wrong subscription.

It was a terrifying moment of clarity.

Re: Gandi loses data, customers told to use their own backups

#340

Post by EnderMB » Fri, Jan 10, 2020, 1:33 AM UTC

danr4 wrote: post #49

EnderMB wrote: post #28

Oof, this Twitter thread looks particularly bad, especially the response from the official Gandi account. https://twitter.com/andreaganduglia/status/12151991477012316... While I appreciate that there are real people behind these companies that are probably having a really rough time right now, the criticism that Gandi are getting as a company is justified - and if Gandi are truly a "no bullshit" company they need to…

Screenshotted in case (when) they delete it https://i.imgur.com/s3R1VVc.png Using memes after permanently losing customer data is extremely disrespectful.

I don't blame the communications rep. From her perspective, she's probably been told what the CEO believes - Gandi lost data, but they never promised backups so it's not a big deal. They responded to someone that is being extremely critical. The rep (Julie) did the right thing and apologised after others criticised her tweet, and also kept the response up to illustrate the mistake. While a meme is bad taste, I can somewhat understand the reaction.

IMO, the blame lies solely with the CEO, because he is still to retract his statement regarding snapshots not being backups (despite their site selling them as backups to the end-user), and for not accepting the fact that for someone controlling business data that creating backups AND regularly testing them via restores is 100% essential. Culture trickles down, and if the CEO only accepts blame and not the reason for the blame then it's a sign that they won't learn from the problem - and that's the biggest red flag you will ever see in ANY business.

I can only see one way back for them that won't taint their reputation completely. They need to:

* Post a full post-mortem of what happened, how it happened, how they fixed it, and what they're going to do to ensure it never happens again.

* Issue a full apology for the problem. Accept full blame, and accept (including the CEO on Twitter) that Gandi failed to follow accepted industry standards.

* Sit down with the engineers that work at Gandi and hear their grievances. While I doubt that their engineers knew this would happen, I'd be willing to bet that there is at least one person there that had raised the lack of off-site backups and no recovery mechanism. That person needs a promotion, and whatever resources needed to fix Gandi.

* Issue a full refund to those that lost data - not a small discount, as already reported. A discount is a kick in the teeth, whereas a full refund is the start of a real apology for failing the customer. If you go for a meal at a restaurant and find broken glass in your food, the first thing the server will do is give you a full refund, no questions asked, regardless of how expensive your parties order was. Gandi need to take the hit, and live to fight another day.