Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire destroys S. Korean government's cloud storage system, no backups available

koreajoongangdaily.joins.com

Post reply on HN
Print view View original

621–630 of 987 posts

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#621

Post by ssl-3 »

belorn wrote: post #404
VirusNewbie wrote: post #300

Earlier quoted context omitted.

[flagged]

As a sysadmin at company that provide fairly sensitive services, I find online cloud backups to be way to slow for the purpose of protecting against something like the server room being destroyed by a fire. Even something like spinning disks at a remote location feel like a risk, as files would need to be copied onto faster disks before services could be restored, and that copying would take precious time during an e…

In one scenario, with offsite backups ("in the clown" or otherwise): "We had a fire at our datacenter, and there will be some downtime while we get things rolling again."

In the other scenario, without offsite backups ("in the clown" or otherwise): "We had a fire at our datacenter, and that shit's just gone."

Neither of these are things that are particularly good to announce, and both things can come with very severe cost, but one of them is clearly worse than the other.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#622

Post by FooBarWidget »

schainks wrote: post #598
StopDisinfo910 wrote: post #240

Earlier quoted context omitted.

The issue here is not refusing to use a foreign third party. That makes sense. The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk. That’s gross mismanagement.

Call me a conspiracy theorist, but this kind of mismanagement is intentional by design so powerful people can hide their dirty laundry.

Never attribute to malice what can be attributed to stupidity.

There was that time when some high profile company's entire Google Cloud account was destroyed. Backups were on Google Cloud too. No off-site backups.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#623

Post by yetihehe »

jeroenhd wrote: post #570
acchow wrote: post #536

Earlier quoted context omitted.

> The issue here is not refusing to use a foreign third party. That makes sense. Encrypt before sending to a third party?

Of course you'd encrypt the data before uploading it to a third party, but there's no reason why that third party should be under control of a foreign government. South Korea has more than one data center they can store data inside of, there's no need to trust other governments sigh every byte of data you've gathered, even if there are no known backdoors or flaws in your encryption mechanism (which I'm sure some gove…

There is a reason that NIST recommends new encryption algorithms from time to time. If you get a copy of ALL government data, in 20 years you might be able to break encryption and get access to ALL government data from 20yr ago, no matter how classified they were, if they were stored in that cloud. Such data might still be valuable, because not all data is published after some period.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#624

Post by procaryote »

Imustaskforhelp wrote: post #389
whartung wrote: post #100

Earlier quoted context omitted.

Testing backups is boring. If you want exciting, test restores!

Hm, care to elaborate. I kinda liked this idea even though I know that it shouldn't make much sense but still lol, would this have any benefits over testing backups other than the excitement lol

The joke is that if you don't test backups you'll end up seeing if they work once you're restoring after a disaster, which is exciting

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#625

Post by tylervigen »

athrowaway3z wrote: post #613

It would be wise for governments to define "backup" as something that is at least 1km away.

Probably farther than that, right? Plenty of natural disasters, including floods and wildfires, can affect an area larger than 1 km.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#626

Post by thanatos519 »

StopDisinfo910 wrote: post #240
kristianc wrote: post #95

The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for extern…

The issue here is not refusing to use a foreign third party. That makes sense. The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk. That’s gross mismanagement.

Backups should be far away, too. Apparently some companies lost everything on 9/11 because their backups were in the other tower.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#627

Post by Lumoscore »

Wow. That is genuinely one of the most terrifying headlines I've read all year.

Seriously, "no backups available" for a national government's main cloud storage? That’s not a simple IT oversight; that’s an epic, unforgivable institutional mistake.

It completely exposes the biggest fear everyone in tech has: putting all the eggs in one big physical basket.

I mean, we all know the rule: if it exists in only one place, it doesn't really exist. If your phone breaks, you still have your photos on a different server, right? Now imagine that basic, common-sense rule being ignored for a country’s central data.

The fire itself is a disaster, but the real catastrophe is the planning failure. They spent millions on a complex cloud system, but they skipped the $5 solution: replicating the data somewhere else—like in a different city, or even just another building across town.

Years of official work, policy documents, and data—just gone, literally up in smoke, because they violated the most fundamental rule of data management. This is a massive, expensive, painful lesson for every government and company in the world: your fancy cloud setup is worthless if your disaster recovery plan is just "hope the building doesn't burn down." It’s an infrastructure nightmare.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#628

Post by reed1234 »

vayup wrote: post #321

A lot of folks are arguing that the real problem is that they refused to use US cloud providers. No, that's not the issue. It's a perfectly reasonable choice to build your own storage infrastructure if it is needed. But the problem is they sacrificed "Availability" in pursuit of security and privacy. Losing your data to natural and man-made disasters is one of the biggest risks facing any storage infrastructure. Any…

Durability is more precise than availability in this context because it is about the data surviving (not avoiding downtime).

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#630

Post by NinjaTrance »

The easy solution would be to use something like Amazon S3 to store documents as objects and let them worry about backup; but governments are worried (and rightly so) about the US government spying on them.

Thus, the not-so-easy-but-arguably-better solution would be to self-host an open source S3-compatible object storage solution.

Are there any good open source alternatives to S3?

Post reply on HN