Live data from Hacker News

Fire destroys S. Korean government's cloud storage system, no backups available

koreajoongangdaily.joins.com

551–560 of 987 posts

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#551
post #491

Goodness, I have over 100TB at home and it cost less than a two or three thousand dollars to put in place. That's like $25 per TB. > The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets. No, the 858TB amounts to under $25k for the government of the 10th largest economy, of one of the most sophisticated countries on the planet, to put in place. Two of those would be less than the price o…

[dead]

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#552

Earlier quoted context omitted.

So right after the investigation was announced, they suddenly scheduled a UPS battery replacement which happened to start a fire big enough to destroy the entire data centre and all data or evidence? Yeah, that's way less suspicious, thanks for clearing that up.

My mind initially went to a government cover-up, but then: > 27th of September 2025, The fire is believed to have been caused while replacing Lithium-ion batteries. The batteries were manufactured by LG, the parent company of LG Uplus (the one that got hacked by the APT). Could the battery firmware have been sabotaged by the hacker to start the fire?

this was a plot in a Mr. Robot episode, heh. Life imitating art?

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#553
post #491

Goodness, I have over 100TB at home and it cost less than a two or three thousand dollars to put in place. That's like $25 per TB. > The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets. No, the 858TB amounts to under $25k for the government of the 10th largest economy, of one of the most sophisticated countries on the planet, to put in place. Two of those would be less than the price o…

~1PB of data, with ingestion at a rate of 12GB per hour, is a tiny amount of data to manage and backup properly for a developed world government. This is silly. Volume clearly should not have been a hinderance.

Backup operations are often complex and difficult - but then again it's been worked on for decades and rigorous protocols exist which can and should be adopted.

"However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained" ... "the G-Drive’s structure did not allow for external backups."

Clearly [in]competence was the single factor here.

This is what happens when you come up with all kind of reasons to do something yourself, which you are not qualified to do, rather than simply paying a vendor to do it for you.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#554
post #524

Earlier quoted context omitted.

Looking at the article, my read (which could be wrong) is that the backup was in the same room as the original.

No. It says that "most systems" in this data center are backed up to separate hardware on a different floor, and then a backup is made at a physically remote location. This particular G-Drive system was not on the standard backup process - it sounds like it was much higher volume than any others, so maybe they couldn't use it. They did have a pilot going to get G-Drive backed up...it was supposed to be scaled up to t…

Wow, rough timing.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#555
What a sad news as a Korean to see a post about Korea at the top of HN during one of the largest Korean holiday.

I can share an anecdote how slow tech adoption is in Korea. It is not exactly about tech in public section but in private companies. I assume public section has slower adoption rate than private ones in general.

Just about a year ago I had a couple of projects with insurance companies. I won't name them but they are the largest ones whose headquarters you can find in the very center of Seoul. They often called me in because I was setting up on-premise servers for the projects. Not to mention that it was hard to understand their choices of database architecture to plug it into the server I was setting up, their data team seemed just incompetent, not knowing what they were doing.

The wildest thing I found was that most office workers seemed to be using windows 2000 to run their proprietary software. To be fair, I like software UIs with a lot of buttons and windows from that era. But alas, I didn't want to imagine myself connecting that legacy software to my then project service. It didn't go that far in the end.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#556

What a sad news as a Korean to see a post about Korea at the top of HN during one of the largest Korean holiday. I can share an anecdote how slow tech adoption is in Korea. It is not exactly about tech in public section but in private companies. I assume public section has slower adoption rate than private ones in general. Just about a year ago I had a couple of projects with insurance companies. I won't name them bu…

Do South Korean companies prefer hosting data on their own servers instead of using Public cloud providers like Azure, AWS, GCP?

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#557

I see some comments about North Korean hacking, so I feel I need to clear up some misconceptions. First, (as you guys have seen) South Korea's IT security track record is not great. Many high-profile commercial sites have been hacked. If a government site was hacked by North Korea, it won't be the first, and while it would be another source of political bickering and finger-pointing, it's likely to blow over in a mon…

You lost me at "Yoon overthrowing the constitution."

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#558
post #13
post #2

> However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost. Yikes. You'd think they would at least have one redundant copy of it all. > erasing work files saved individually by some 750,000 civil servants > 30 gigabytes of storage per person That's 22,500 terabytes, about 50 Backblaze storage pods. Or even just mir…

It's even worse. According to other articles [1], the total data of "G drive" was 858 TB. It's almost farcical to calculate, but AWS S3 has pricing of about $0.023/GB/month, which means the South Korean government could have reliable multi-storage backup of the whole data at about $20k/month. Or about $900/month if they opted for "Glacier deep archive" tier ($0.00099/GB/month). They did have backup of the data ... in…

>AWS S3 has pricing of about $0.023/GB/month, which means ... about $20k/month

or outright buying hardware capable of storing 850TB for the same $20K one time payment. Gives you some perspective on how overpriced AWS is.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#559

Earlier quoted context omitted.

They are overwhelmingly whitelabeled providers. For example, Samsung SDI Cloud (the largest "Korean" cloud) is an AWS white label. Korea is great at a lot of engineering disciplines. Sadly, software is not one of them, though it's slowly changing. There was a similar issue a couple years ago where the government's internal intranet was down a couple days because someone deployed a switch in front of outbound connecti…

> Korea is great at a lot of engineering disciplines. Sadly, software is not one of them I disagree. People say the same about Japan and Taiwan (and Germany). IMHO, they are overlooking the incredible talents in embedded programming. Think of all of the electronics (including automobiles) produced in those countries.

Embedded electronics, including from those countries, does not have an enviable reputation. :(

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#560
post #502
post #120

Earlier quoted context omitted.

Agree completely that it's absolute wild to run such a system without backups. But at this point no government should keep critical data on foreign cloud storage.

It's 2025. Encryption is a thing now. You can store anything you want on foreign cloud storage. I'd give my backups to the FSB.

> I'd give my backups to the FSB.

Until you need them - like with the article here ;) - then the FSB says "only if you do these specific favours for us first...".

Post reply on HN