Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire destroys S. Korean government's cloud storage system, no backups available

koreajoongangdaily.joins.com

Post reply on HN
Print view View original

751–760 of 987 posts

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#751

Post by johannes1234321 »

fauigerzigerk wrote: post #702
zestyping wrote: post #680

Earlier quoted context omitted.

> This is incredible. Government telling me how to backup my data. Incredible. No more incredible than the government telling you that you need liability insurance in order to drive a car. Do you think that is justifiable?

The difference is that you cannot choose who you're sharing a road with while you can usually choose your IT service providers. You could, for instance, choose a cheaper provider and make your own backups or simply accept that you could lose your data. Where people have little or no choice (e.g government agencies, telecoms, internet access providers, credit agencies, etc) or where the blast radius is exceptionally w…

Losing data is mostly(*) fine if you are a small business. If a major bank loses it's data it is a major problem as it may impact a huge number of customers and an existential way, when all money is "gone"

(*) From state's perspective there is still a problem: tax audits, bad if everybody avoids them by "accidental" data loss

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#752

Post by INTPenis »

kristianc wrote: post #95

The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for extern…

Dude, the issues go wayyy beyond opting for selfhosting rather than US clouds.

We use selfhosting, but we also test our fire suppression system every year, we have two different DCs, and we use S3 backups out of town.

Whoever runs that IT department needs to be run out of the country.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#753

Post by rotis »

m3047 wrote: post #42

Mindblowing. Took a walk. All I can say is that if business continues "as usual" and the economy and public services continue largely unaffected then either there were local copies of critical documents, or you can fire a lot of those workers; either one of those ways the "stress test" was a success.

The fire started on 26th September and news about it reached HN only now. I think this is telling how disruptive for South Korea daily life this accident really was.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#754

Post by niffydroid »

jftr wrote: post #358
msbhvn wrote: post #122

Earlier quoted context omitted.

Woah, read the timeline at the top of this. The fire happened the very day the government ordered onsite inspection was supposed to start due to Chinese/NK hacking.

Phrack's timeline may read like it, but it wasn't an onsite inspection due to hacking, but a scheduled maintenance to replace the overdue UPS, hence battery-touching involved. Even the image they linked just says "scheduled maintenance."

https://www.ispreview.co.uk/index.php/2025/09/openreach-give...

Recently in the UK a major communication company had issues with batteries

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#755

Post by axus »

jeroenhd wrote: post #574
yongjik wrote: post #13

Earlier quoted context omitted.

It's even worse. According to other articles [1], the total data of "G drive" was 858 TB. It's almost farcical to calculate, but AWS S3 has pricing of about $0.023/GB/month, which means the South Korean government could have reliable multi-storage backup of the whole data at about $20k/month. Or about $900/month if they opted for "Glacier deep archive" tier ($0.00099/GB/month). They did have backup of the data ... in…

AWS? Linus Tech Tips has run multiple petabyte servers in their server closet just for sponsor money and for the cool of it. No need to outsource your national infrastructure to foreign governments, a moderate (in government terms) investment in a few racks across the country could've replicated everything for maybe half a year's worth of Amazon subscription fees.

But then they will depend on the security people at both sides to agree on the WAN configuration. Easier to let everything burn in a fire and rebuild from scratch.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#756

Post by sbinnee »

niutech wrote: post #668
gkanai wrote: post #659

Earlier quoted context omitted.

Back when I worked for Mozilla, I had the chance to go to Seoul to meet with various companies and some governmental ministries. This was when Korean banks and ecommerce sites required Internet Explorer and Active-X controls for secure transactions. This meant that MacOS users or Linux users couldnt do secure transactions in Korea without emulating Win/IE.

What was the outcome of these meetings? Have they switched to Firefox?

They settled down with chromium based browsers. Microsoft was pushing Edge and Naver, the largest Korean search engine company, also developed Whale browser based on chromium.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#757

Post by waste_monk »

sleepybrett wrote: post #381
rawgabbit wrote: post #367

Earlier quoted context omitted.

How large is this UPS that a fire can bring down all 96 servers? This story is really unbelievable.

depends on how many batteries were in the facility, if one goes up chances are the rest go too. Can halon systems not put out lithium fires?

>Can halon systems not put out lithium fires?

As the other commenter said, Halon hasn't been a thing for a fair while, but inert gas fire suppression systems in general are still popular.

I would expect it wouldn't be sufficient for a lithium ion battery fire - you'd temporarily displace the oxygen, sure, but the conditions for fire would still exist - as soon as enough nitrogen (or whatever suppressant gas is in use) dissipates, it'd start back up again.

Also as I understand thermal runaway is self-sustaining, since the lithium ion batteries have a limited capacity to provide their own oxygen (something to do with the cathode breaking down?), so it might continue burning even while the area is mostly flooded with inert gas.

I believe it would be similar to an EV car fire, that is, you'd have to flood the area with water and wait for it to cool down enough that thermal runaway stops. Maybe they can do better these days with encapsulating agents but I'd still expect the rack housing the UPS to be a write-off.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#758

Post by 100721 »

xoa wrote: post #494
100721 wrote: post #360

Earlier quoted context omitted.

As someone who’s fairly tech-literate but has a big blind spot in cryptography, I’d love to hear any suggestions you have for articles, blog posts, or smaller books on the topic! My (rudimentary, layman) understanding is that encryption is almost like a last line of defense and should never be assumed to be unbreakable. You sound both very knowledgeable on the topic, and very confident in the safety of modern encrypt…

Whew, that's actually a hard one! It's been long enough since I was getting into it that I'm not really sure what's the best present path on it. In terms of books, JP Aumasson's "Serious Cryptography" got a 2nd edition not too long ago and the first edition was good. Katz & Lindell's "Modern Cryptography" and Hoffstein's "Introduction to Mathematical Cryptography" are both standard texts that I think a lot of courses…

Wow, thank you for this detailed reply! I’ll be checking out some of those resources at lunch today :)

I didn’t take your comment as a dig at all. I’m honestly a little surprised myself that I’ve made it this far with such a flawed understanding.

> Did you think they could do it anyway so trying to pass a law to force backdoors to be made is a cover for existing capabilities, or what?

I had to do some quick reading on the ChatControl proposal in the EU.

I see it along the lines of, if they really needed to target someone in particular (let’s not get into who “deserves” to be targeted), then encryption would only be an obstacle for them to have to overcome. But, for the great majority of traffic - like our posts being submitted to HN - the effort of trying to break the encryption (eg, dedicating a few months of brute force effort across multiple entire datacenters) simply isn’t worth it. In many other scenarios, bypassing the encryption is a lot more practical, like that one operation where I believe it was the FBI waited for their target to unlock his laptop - decrypting the drive - in a public space, and then they literally grabbed the laptop and ran away with it.

The ChatControl proposal sounds like it aims to bypass everyone’s encryption, making it possible to read and process all user data that goes across the wire. I would never be in support of something like that, because it sounds like it sets up a type of backdoor that is always present, and always watching. Like having a bug planted in your apartment where everything you say is monitored by some automated flagging system, just like in 1984.

If a nation state wants to spend the money to dedicate multiple entire datacentres to brute forcing my encrypted communications, achieving billions of years of compute time in the span of a few months or whatever, I’m not a fan but at least it would cost them an arm and a leg to get those results. The impracticality of such an approach makes it so that they don’t frivolously pursue such efforts.

The ability to view everyone’s communications in plaintext is unsettling and seems like it’s just waiting to be abused, much in the same way that the United States’ PRISM was (and probably is still being) abused.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#759

Post by bell-cot »

catlifeonmars wrote: post #581
charlieyu1 wrote: post #339

Earlier quoted context omitted.

You don’t need cloud when you have the data centre, just backups in physical locations somewhere else

Others have pointed out: you need uptime too. So a single data center on the same electric grid or geographic fault zone wouldn’t really cut it. This is one of those times where it sucks to be a small country (geographically).

> so a single data center on the same electrical grid or geographic...

Yes, but your backup DC's can have diesel generators and a few weeks of on-site fuel. It has some quakes - but quake-resistant DC's exist, and SK is big enough to site 3 DC's at the corners of an equilateral triangle with 250km edges. Similar for typhoons. Invading NK armies and nuclear missiles are tougher problems - but having more geography would be of pretty limited use against those.

Post reply on HN