Fire destroys S. Korean government's cloud storage system, no backups available • Hacker News Board

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#901

Post by tsimionescu » Mon, Oct 06, 2025, 5:42 PM UTC

waterTanuki wrote: post #506

kazinator wrote: post #280

Earlier quoted context omitted.

You and I can encrypt our data before saving it into the cloud, because we have nothing of value or interest to someone with the resources of a state. Sometimes sensitive data at the government level has a pretty long shelf life; you may want it to remain secret in 30, 50, 70 years.

I don't see how this is any different than countries putting significant portions of their gold & currency reserves in the NY Federal Reserve Bank. If for some reason the U.S. just decided to declare "Your monies are all mine now" the effects would be equally if not more devastating than a data breach.

The difference is that there are sometimes options to recover the money, and at least other countries will see and know that this happened, and may take some action.

A data breach, however, is completely secret - both from you and from others. Another country (not even necessarily the one that is physically hosting your data) may have access to your data, and neither you nor anyone else would necessarily know.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#902

Post by ycombinatrix » Mon, Oct 06, 2025, 5:43 PM UTC

mastax wrote: post #141

yongjik wrote: post #13

Earlier quoted context omitted.

It's even worse. According to other articles [1], the total data of "G drive" was 858 TB. It's almost farcical to calculate, but AWS S3 has pricing of about $0.023/GB/month, which means the South Korean government could have reliable multi-storage backup of the whole data at about $20k/month. Or about $900/month if they opted for "Glacier deep archive" tier ($0.00099/GB/month). They did have backup of the data ... in…

I made an 840TB storage server last month for $15,000.

840TB before or after configuring RAID?

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#903

Post by BizarroLand » Mon, Oct 06, 2025, 5:43 PM UTC

mliezun wrote: post #712

quantumsequoia wrote: post #327

Earlier quoted context omitted.

Most people overestimate the prevalence of malice, und underestimate the prevalence of incompetence

What do you make of this? The guy who was in charge of restoring the system was found dead https://www.thestar.com.my/aseanplus/aseanplus-news/2025/10/...

My guess would be that either he felt it was such a monumental cockup that he had to off himself or his bosses thought it was such a monumental cockup that they had to off him.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#904

Post by 4WIW » Mon, Oct 06, 2025, 5:59 PM UTC

The board of directors should now fire the management over such as gross mismanagement. Then, the board of directors should be fired for not proactively requiring backups.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#905

Post by HardCodedBias » Mon, Oct 06, 2025, 6:03 PM UTC

eCa wrote: post #120

kristianc wrote: post #95

The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for extern…

Agree completely that it's absolute wild to run such a system without backups. But at this point no government should keep critical data on foreign cloud storage.

Why not?

Has there been any interruption in service?

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#906

Post by arcfour » Mon, Oct 06, 2025, 6:14 PM UTC

StopDisinfo910 wrote: post #240

kristianc wrote: post #95

The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for extern…

The issue here is not refusing to use a foreign third party. That makes sense. The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk. That’s gross mismanagement.

I very seriously doubt that the US cares about South Korea's deepest, darkest secrets that much, if at all.

Not using a cloud provider is asinine. You can use layered encryption so the expected lifetime of the cryptography is beyond the value of the data...and the US government themselves store data on all 3 of them, to my knowledge.

I say US because the only other major cloud providers I know of are in China, and they do have a vested interest in South Korean data, presumably for NK.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#907

Post by arcfour » Mon, Oct 06, 2025, 6:17 PM UTC

raxxorraxor wrote: post #730

acchow wrote: post #536

Earlier quoted context omitted.

> The issue here is not refusing to use a foreign third party. That makes sense. Encrypt before sending to a third party?

Why make yourself dependent on a foreign country for your own sensitive data? You have to integrate the special software requirements to any cloud storage anyway and hosting a large amount of files isn't an insurmountable technical problem. If you can provide the minimal requirements like backups, of course.

Presumably because you aren't capable of building what that foreign country can offer you yourself.

Which they weren't. And here we are.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#908

Post by sllabres » Mon, Oct 06, 2025, 6:26 PM UTC

linksnapzz wrote: post #863

palmotea wrote: post #832

Earlier quoted context omitted.

Before 9/11, most DR (disaster recovery) sites were in Jersey City, NJ just across the river from their main offices in WFC or WTC, or roughly 3-5 miles away. After 9/11, the financial industry adopted a 50+ miles rule. IIRC, multiple IBM mainframes can be setup so they run and are administered as a single system for DR, but there are distance limits.

A Geographically-Dispersed Parallel Sysplex for z\OS mainframes, which IBM has been selling since the '90s, can have redundancy out to about 120 miles. At a former employer, we used a datacenter in East Brunswick NJ that had mainframes in sysplex with partners in lower manhattan.

If you have to mirror synchronously the _maximum_ distances for other systems (e.g. storage mirroring with NetApp SnapMirror Synchronous, IBM PPRC, EMC SRDF/S) are all in this range.

But an important factor is, that performance will degrade with every microsecond latency added as the active node for the transaction will have to wait for the acknowledgement of the mirror node (~2*RTT). You can mirror synchronously that distance, but the question is if you can accept the impact.

That's not to say that one shouldn't create a replica in this case. If necessary, synchronize synchronous to a nearby DC and asynchrone to a remote one.

For sure we only know the sad consequences.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#909

Post by xp84 » Mon, Oct 06, 2025, 6:38 PM UTC

qzw wrote: post #786

stego-tech wrote: post #773

Earlier quoted context omitted.

This. Speaking specifically from the IT side of things, an employer or customer refusing to do backups is the biggest red flag I can get, an immediate warning to run the fuck away before you get blamed for their failure, stego-tech kind of situation. That being said, I can likely guess where this ends up going: * Current IT staff and management are almost certainly scapegoated for “allowing this to happen”, despite t…

I abhor the general trend of governments outsourcing everything to private companies, but in this case, a technologically advanced country’s central government couldn’t even muster up the most basic of IT practices, and as you said, accountability will likely not rest with the people actually responsible for this debacle. Even a nefarious cloud services CEO couldn’t dream up a better sales case for the wholesale outs…

I'm with you. It's really sad that this provides such a textbook case of why not to own your own infrastructure.

Practically speaking, I think a lot of what is offered by Microsoft, Google, and the other big companies that are selling into this space is vastly overpriced and way too full of lock-in, taking this stuff in-house without sufficient knowhow and maturity is even more foolish.

It's like not hiring professional truck drivers, but instead of at least people who can basically drive a truck, hiring someone who doesn't even know how to drive a car.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#910

Post by xp84 » Mon, Oct 06, 2025, 6:41 PM UTC

SEJeff wrote: post #737

flumpcakes wrote: post #673

Earlier quoted context omitted.

Companies big enough will lay the fibre. 50-100 miles of fibre isn't much if you are a billion dollar business. Even companies like BlackRock who had their own datacenters have since taken up Azure. 50 miles latency is negligible, even for databases.

In the US, dark fiber will run you around 100k / mile. Thats expensive for anyone even if they can afford it. I worked in HFT for 15 years and we had tons of it.

So that's 5 million bucks for 50 miles? If there are other costs not being accounted for, like paying for the right-of-way that's one thing, but I would think big companies or in this case, a national government, could afford that bill.