Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire destroys S. Korean government's cloud storage system, no backups available

koreajoongangdaily.joins.com

Post reply on HN
Print view View original

531–540 of 987 posts

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#531

Post by PapaPalpatine »

naruhodo wrote: post #425
danudey wrote: post #399

Earlier quoted context omitted.

So right after the investigation was announced, they suddenly scheduled a UPS battery replacement which happened to start a fire big enough to destroy the entire data centre and all data or evidence? Yeah, that's way less suspicious, thanks for clearing that up.

My mind initially went to a government cover-up, but then: > 27th of September 2025, The fire is believed to have been caused while replacing Lithium-ion batteries. The batteries were manufactured by LG, the parent company of LG Uplus (the one that got hacked by the APT). Could the battery firmware have been sabotaged by the hacker to start the fire?

That's exactly where my mind went!

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#532

Post by kepano »

When I visited the National Museum of Korea in Seoul, one of my favorite parts was exploring the exhibit dedicated to the backing up state data — via calligraphy, letterpress, and stone carving.

> "The Veritable Records of the Joseon Dynasty, sometimes called sillok (실록) for short, are state-compiled and published records, documenting the reigns of the kings of the Joseon dynasty in Korea. Kept from 1392 to 1865, they comprise 1,893 volumes and are thought to be the longest continual documentation of a single dynasty in the world."

> "Beginning in 1445, they began creating three additional copies of the records, which they distributed at various locations around Korea for safekeeping."

https://en.wikipedia.org/wiki/Veritable_Records_of_the_Joseo...

After the Japanese and Qing invasions of Japan, King Hyeonjong (1659–1675) started a project to collect calligraphy works written by preceding Joseon kings and carve them into stone.

It's somewhat surprising that these values didn't continue to persist in the Korean government.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#533

Post by marcoglauser »

somenameforme wrote: post #530
croes wrote: post #437

Earlier quoted context omitted.

That’s why Microsoft can't guarantee data sovereignty https://news.ycombinator.com/item?id=45061153

He obviously meant encrypting before uploading. At that point it doesn't matter who's holding your data or what they try to do with it.

It still matters who holds your data. Yes they can't read it, but they can hold it ransom. What if the US decides it wants to leverage the backups in tariff negotiations or similar? Not saying this would happen, but as a state level actor, you have to prepare for these eventualities.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#534

Post by somenameforme »

marcoglauser wrote: post #533
somenameforme wrote: post #530

Earlier quoted context omitted.

He obviously meant encrypting before uploading. At that point it doesn't matter who's holding your data or what they try to do with it.

It still matters who holds your data. Yes they can't read it, but they can hold it ransom. What if the US decides it wants to leverage the backups in tariff negotiations or similar? Not saying this would happen, but as a state level actor, you have to prepare for these eventualities.

That's why you backup to numerous places and in numerous geopolitical blocs. Single points of failure are always a bad idea. You have to create increasingly absurd scenarios for there to be a problem.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#535

Post by burnt-resistor »

DR/BCP fail. The old adage companies that lose all of their data typically go out of business within 6 months I guess doesn't apply when it's the government.

At a minimum, they could've stored the important bits like financial transactions, personnel/HR records, and asset inventory database backups to Tarsnap [0] and shoved the rest in encrypted tar backups to a couple of different providers like S3 Glacier and/or Box.

Business impact analysis (BIA) is a straightforward way to assessing risks of probability of event * cost to recover from event = approximate budget for spending on mitigation.

And, PSA: test your backups and DR/BCP runbooks periodically!

0. https://www.tarsnap.com

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#536

Post by acchow »

StopDisinfo910 wrote: post #240
kristianc wrote: post #95

The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for extern…

The issue here is not refusing to use a foreign third party. That makes sense. The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk. That’s gross mismanagement.

> The issue here is not refusing to use a foreign third party. That makes sense.

Encrypt before sending to a third party?

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#537

Post by efitz »

belorn wrote: post #404
VirusNewbie wrote: post #300

Earlier quoted context omitted.

[flagged]

As a sysadmin at company that provide fairly sensitive services, I find online cloud backups to be way to slow for the purpose of protecting against something like the server room being destroyed by a fire. Even something like spinning disks at a remote location feel like a risk, as files would need to be copied onto faster disks before services could be restored, and that copying would take precious time during an e…

For truly colossal amounts of data, fedex has more bandwidth than fiber. I don’t know if any cloud providers will send you your stuff on physical storage, but most will allow you to send your stuff to them on physical storage- eg AWS snowball.

There are two main reasons why people struggle with cloud restore:

1. Not enough incoming bandwidth. The cloud’s pipe is almost certainly big enough to send your data to you. Yours may not be big enough to receive it.

2. Cheaping out on storage in the cloud. If you want fast restores, you can’t use the discount reduced redundancy low performance glacier tier. You will save $$$ right until the emergency where you need it. Pay for the flagship storage tier- normal AWS S3, for example- or splurge and buy whatever cross-region redundancy offering they have. Then you only need to worry about problem #1.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#538

Post by burnt-resistor »

vbezhenar wrote: post #320
Ekaros wrote: post #136

Earlier quoted context omitted.

When you start to do math, hard drive are cheap when you go for capacity and not performance. 0.00099*1000 is 0.99. So about 12$ a year. Now extrapolate something like 5 year period or 10 year period. And you get to 60 to 120$ for TB. Even at 3 to 5x redundancy those numbers start to add up.

S3 does not spend 3x drives to provide redundancy. Probably 20% more drives or something like that. They split data to chunks and use erasure coding to store them in multiple drives with little overhead.

And S3 RRS and Glacier do even less.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#539

Post by BonoboIO »

crmd wrote: post #208

I would love to know how a fire of this magnitude could happen in a modern data center.

Easy, some electronic fault and look at OVH with WOODEN FLOOR and bad management decisions. But of course the servers had automatic backups … in the datacenter in the same building. A few companies lost EVERYTHING and had to close, because of this.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#540

Post by TulliusCicero »

kristianc wrote: post #95

The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for extern…

> G-Drive’s structure did not allow for external backups

Ha! "Did not allow" my ass. Let me translate:

> We didn't feel like backing anything up or insisting on that functionality.

Post reply on HN