Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"Hetzner decided to cancel our account and terminate all servers"

mastodon.social

Post reply on HN
Print view View original

151–160 of 401 posts

Re: "Hetzner decided to cancel our account and terminate all servers"

#151

Post by kelnos »

ambicapter wrote: post #142
diggan wrote: post #22

Earlier quoted context omitted.

Another great question is "When did you last try to restore from a backup?" which usually is answered with "It's the built-in tooling, why would we assume it's broken?" or similar. Then fast-forward some months/years, and they try to restore from backups only to realize the backups never actually backed up what they cared about.

One thing I've never figured out is, what is the difference between backups and replication? And, does restoring from backups always mean losing more _recent_ data than replication?

One problem with replication is if the disaster is that all the data has been deleted, that deleted state will get propagated to the replica, so you will still have no data.

But yes, if the problem is simply that the main setup is down, replication will often give you a more (or even completely) up-to-date copy than a daily backup will.

Re: "Hetzner decided to cancel our account and terminate all servers"

#152

Post by travisby »

ambicapter wrote: post #142
diggan wrote: post #22

Earlier quoted context omitted.

Another great question is "When did you last try to restore from a backup?" which usually is answered with "It's the built-in tooling, why would we assume it's broken?" or similar. Then fast-forward some months/years, and they try to restore from backups only to realize the backups never actually backed up what they cared about.

One thing I've never figured out is, what is the difference between backups and replication? And, does restoring from backups always mean losing more _recent_ data than replication?

For hardware failure, replication is the bees knees and indeed means you'll lose less (no? depending on your replication settings) data.

But, backups will help if you replicated _bad data_, or more accurately _data changes_.

You can restore from backup if you accidentally ran `DELETE FROM foo;`, where replication will not help!

(Insert cryptolocker type viruses, bugs, human query mistakes, etc).

Re: "Hetzner decided to cancel our account and terminate all servers"

#153

Post by 0cf8612b2e1e »

SamWhited wrote: post #131
ratg13 wrote: post #113

Earlier quoted context omitted.

I didn't default to them, but did start a new project on their infrastructure. They deleted all of my data a month in due to not beleiving my name was real, and without even bothering to contact me to verify anything. They deleted my backups as well because I was dumb enough to keep them under the same account. I learned a valuable lesson the hard way and have improved my methods as a result, but sad that it cost me…

I had actually forgotten about this, I had a friend who had the exact same thing happen (dropped because "you have to use real names" or whatever, but they did use their real name, and it wasn't even anything suspicious or weird [not that that should matter], they just have a vaguely common for Eastern Europe sounding name :S)

That feels like a preposterous automated policy. How would you design rules for what is a real name? At least, raise it for human review and some kind of manual validation before nuking an account.

Re: "Hetzner decided to cancel our account and terminate all servers"

#154

Post by kopirgan »

SamWhited wrote: post #131
ratg13 wrote: post #113

Earlier quoted context omitted.

I didn't default to them, but did start a new project on their infrastructure. They deleted all of my data a month in due to not beleiving my name was real, and without even bothering to contact me to verify anything. They deleted my backups as well because I was dumb enough to keep them under the same account. I learned a valuable lesson the hard way and have improved my methods as a result, but sad that it cost me…

I had actually forgotten about this, I had a friend who had the exact same thing happen (dropped because "you have to use real names" or whatever, but they did use their real name, and it wasn't even anything suspicious or weird [not that that should matter], they just have a vaguely common for Eastern Europe sounding name :S)

Guess Mr. Phuc dat Bich from Hanoi needn't bother applying.

Wonder what's the algorithm they use to know a "real name"

Re: "Hetzner decided to cancel our account and terminate all servers"

#155

Post by no_wizard »

KingOfCoders wrote: post #19

Whenever I ask a CTO if they have a backup (or plan-B) they say we're on AWS, we backup there and they will never go down as a company. And then I ask them what they do when their account gets shut (e.g. because they are selling something bad on Amazon and have the same phone number as the company account?) Or the instance some years ago where GCP closed because someone had wrongly classified image on their drive? Yo…

When should one start doing this though, in a companies life cycle?

What is the most reasonable point that meets the criteria of 'as soon as possible'?

Because I imagine out of the gate doing this could be a net negative, not a net positive.

On the other hand, I'm not sufficiently well versed enough on the absolute latest devops techniques that may make this whole thing trivial, but I thought all the major cloud providers had just enough quirks in their Terraform support you can't write once standup / deploy anywhere

Re: "Hetzner decided to cancel our account and terminate all servers"

#156

Post by tracerbulletx »

KingOfCoders wrote: post #19

Whenever I ask a CTO if they have a backup (or plan-B) they say we're on AWS, we backup there and they will never go down as a company. And then I ask them what they do when their account gets shut (e.g. because they are selling something bad on Amazon and have the same phone number as the company account?) Or the instance some years ago where GCP closed because someone had wrongly classified image on their drive? Yo…

There should just be a legal duty placed on cloud providers to not do this. Nobody would expect you to hold a second redundant commercial lease for your offices or retail location.

Re: "Hetzner decided to cancel our account and terminate all servers"

#157

Post by dilyevsky »

aurareturn wrote: post #4

Hang on, Hetzner literally deleted all their data without warning? That’s actually insane and business killing. Both for Hetzner’s reputation and potentially for their customer.

Yep, this is common with Hetzner and has been the case since forever. Unfortunately all the comments even suggesting that Hetzner is not good for running serious scaled businesses for this reason and many others usually get downvoted to oblivion and remain hidden.

Re: "Hetzner decided to cancel our account and terminate all servers"

#158

Post by sidewndr46 »

diggan wrote: post #22
KingOfCoders wrote: post #19

Whenever I ask a CTO if they have a backup (or plan-B) they say we're on AWS, we backup there and they will never go down as a company. And then I ask them what they do when their account gets shut (e.g. because they are selling something bad on Amazon and have the same phone number as the company account?) Or the instance some years ago where GCP closed because someone had wrongly classified image on their drive? Yo…

Another great question is "When did you last try to restore from a backup?" which usually is answered with "It's the built-in tooling, why would we assume it's broken?" or similar. Then fast-forward some months/years, and they try to restore from backups only to realize the backups never actually backed up what they cared about.

my dad told me about this customer that had a server that made automatic backups each Sunday night. The backup script would backup all the data then eject the tape so the manager could put it in the vault and rotate in the other one from the vault.

When the hard drive failed, they restored the customer to the latest backup. Which was the tape still sitting in the tape drive in the server. It was from the first Sunday night after the system was installed years ago

Re: "Hetzner decided to cancel our account and terminate all servers"

#159

Post by buran77 »

milesward wrote: post #143
endgame wrote: post #37

Earlier quoted context omitted.

https://cloud.google.com/blog/products/infrastructure/detail... Google Cloud accidentally wiped an Australian super[annuation] (pension) fund's entire cloud deployment earlier this year. I think that if you really want durable backups, they have to be reducible to object storage and put in someone else's cloud.

... not quite. I worked directly with the folks involved on getting more RCA details public. This customer used a single product on GCP, a specific type of VMware hosting, and the "subscription" to that product failed, which turned those resources off. It's more like turning off all their VM's, rather than deleting their entire account, identities, access structures, etc.

The reporting on that was a bid muddy with Google and Unisuper officially saying different things in different places. Regardless, calling it "more like turning off all their VM's" sounds like heavily downplaying the reality. The downtime alone confirms it was way more than that.

From their joint statement [0]:

> when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.

> an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications.

> UniSuper had backups in place with an additional service provider. These backups have minimised data loss

Strangely enough on this last point a Google blog post [1] says:

> This incident did not impact: The customer’s data backups stored in Google Cloud Storage (GCS) in the same region.

[0] https://www.unisuper.com.au/about-us/media-centre/2024/a-joi...

[1] https://cloud.google.com/blog/products/infrastructure/detail...

Re: "Hetzner decided to cancel our account and terminate all servers"

#160

Post by sidewndr46 »

travisby wrote: post #152
ambicapter wrote: post #142

Earlier quoted context omitted.

One thing I've never figured out is, what is the difference between backups and replication? And, does restoring from backups always mean losing more _recent_ data than replication?

For hardware failure, replication is the bees knees and indeed means you'll lose less (no? depending on your replication settings) data. But, backups will help if you replicated _bad data_, or more accurately _data changes_. You can restore from backup if you accidentally ran `DELETE FROM foo;`, where replication will not help! (Insert cryptolocker type viruses, bugs, human query mistakes, etc).

I imagine in that scenario the engineering team can develop inter-dimensional travel, then travel to a universe in which that command was never executed. They bring the data back and restore the database.
Post reply on HN