Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire destroys S. Korean government's cloud storage system, no backups available

koreajoongangdaily.joins.com

Post reply on HN
Print view View original

771–780 of 987 posts

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#771

Post by lambdaone »

dkga wrote: post #741
cornholio wrote: post #606

Earlier quoted context omitted.

Because it lowers the threshold for a total informational compromise attack from "exfiltrate 34PB of data from secure govt infrastructure" down to "exfiltrate 100KB of key material". You can get that out over a few days just by pulsing any LED visible from outside an air-gapped facility.

Wait what?

There are all sorts of crazy ways of getting data out of even air-gapped machines, providing you are willing to accept extremely low data rates to overcome attenuation. Even with million-to-one signal-to-noise ratio, you can get significant amounts of key data out in a few weeks.

Jiggling disk heads, modulating fan rates, increasing and decreasing power draw... all are potential information leaks.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#772

Post by xoa »

jacquesm wrote: post #516
xoa wrote: post #476

Earlier quoted context omitted.

> I can't reveal any details but it was a lot more than just a given storage point Sorry, not brain not really clicking tonight and used lazy imprecise terminology here, been a long one. But what I meant by "store point" was any single data repository that can be interacted with as a unit, regardless of implementation details, that's part of a holistic data storage strategy. So in this case the entirety of IBM would…

Ah ok, clear. Thank you for the clarification. Some more interesting details: the initial fault was triggered by a test of a fire suppression system, that would have been recoverable. But someone thought they were exceedingly clever and they were going to fix this without any downtime and that's when a small problem became a much larger one, more so when they found out that their backups were incomplete. I still wond…

Thanks in turn for the details, always fascinating (and useful for lessons... even if not always for the party in question dohoho) to hear a touch of inside baseball on that kind of incident.

>But someone thought they were exceedingly clever and they were going to fix this without any downtime and that's when a small problem became a much larger one

The sentence "and that's when a small problem became a big problem" comes up depressingly frequently in these sorts of post mortems :(. Sometimes sort of feels like, along all the checklists and training and practice and so on, there should also simply be the old Hitchhiker's Guide "Don't Panic!" sprinkled liberally around along with a dabbing of red/orange "...and Don't Be Clever" right after it. We're operating in alternate/direct law here folks, regular assumptions may not hold. Hit the emergency stop button and take a breath.

But of course management and incentive structures play a role in that too.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#773

Post by stego-tech »

StopDisinfo910 wrote: post #240
kristianc wrote: post #95

The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for extern…

The issue here is not refusing to use a foreign third party. That makes sense. The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk. That’s gross mismanagement.

This. Speaking specifically from the IT side of things, an employer or customer refusing to do backups is the biggest red flag I can get, an immediate warning to run the fuck away before you get blamed for their failure, stego-tech kind of situation.

That being said, I can likely guess where this ends up going:

* Current IT staff and management are almost certainly scapegoated for “allowing this to happen”, despite the program in question (G-DRIVE) existing since 2017 in some capacity.

* Nobody in government will question sufficiently what technical reason is/was given to justify the lack of backups and why that was never addressed, why the system went live with such a glaring oversight, etc, because that would mean holding the actual culprits accountable for mismanagement

* Everyone involved is unlikely to find work again anytime soon once names are bandied about in investigations

* The major cloud providers will likely win several contracts for “temporary services” that in actuality strip the sovereignty the government had in managing its own system, even if they did so poorly

* Other countries will use this to justify outsourcing their own sovereign infrastructure to private enterprise

This whole situation sucks ass because nothing good is likely to come of this, other than maybe a handful of smart teams lead by equally competent managers using this to get better backup resources for themselves.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#774

Post by BonitaPersona »

phatfish wrote: post #736

"The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets" Just so we can all visualise this in an understandable way, if laid end-to-end how many times round the world would the A4 sheets go? And what is their total area in football fields?

Actual football fields please, the International Standard Unit footbal field, used in SI countries.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#775

Post by withinboredom »

laserlight wrote: post #679
FooBarWidget wrote: post #622

Earlier quoted context omitted.

Never attribute to malice what can be attributed to stupidity. There was that time when some high profile company's entire Google Cloud account was destroyed. Backups were on Google Cloud too. No off-site backups.

> Never attribute to malice what can be attributed to stupidity. Any sufficiently advanced malice is indistinguishable from stupidity. I don't think there's anything that can't be attributed to stupidity, so the statement is pointless. Besides, it doesn't really matter naming an action stupidity, when the consequences are indistinguishable from that of malice.

I know of one datacenter that burned down because someone took a dump before leaving for the day, the toilet overflowed, then flooded the basement, and eventually started an electrical fire.

I'm not sure you could realistically explain that as anything. Sometimes ... shit happens.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#776

Post by ants_everywhere »

WCSTombs wrote: post #617
nurumaik wrote: post #592

Earlier quoted context omitted.

Why not?

As of today, there's no way to prove the security of any available cryptosystem. Let me say that differently: for all we know, ALL currently available cryptosystems can be easily cracked by some unpublished techniques. The only sort-of exception to that requires quantum communication, which is nowhere near practicability on the scale required. The only evidence we have that the cryptography that we commonly use is ac…

> The only evidence we have that the cryptography that we commonly use is actually safe is that it's based on "hard" math problems that have been studied for decades or longer by mathematicians without anyone being able to crack them.

Adding to this...

Most crypto I'm aware of implicitly or explicitly assumes P != NP. That's the right practical assumption, but it's still an major open math problem.

If P = NP then essentially all crypto can be broken with classical (i.e. non-quantum) computers.

I'm not saying that's a practical threat. But it is a "known unknown" that you should assign a probability to in your risk calculus if you're a state thinking about handing over the entirety of your encrypted backups to a potential adversary.

Most of us just want to establish a TLS session or SSH into some machines.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#777

Post by abujazar »

cornholio wrote: post #606
nurumaik wrote: post #592

Earlier quoted context omitted.

Why not?

Because it lowers the threshold for a total informational compromise attack from "exfiltrate 34PB of data from secure govt infrastructure" down to "exfiltrate 100KB of key material". You can get that out over a few days just by pulsing any LED visible from outside an air-gapped facility.

On which TV show?

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#778

Post by whatevaa »

lima wrote: post #143
whatevaa wrote: post #121

Earlier quoted context omitted.

Yeah, I heard that consumer clouds are only locally redundant and there aren't even backups. So big DC damage could result in data loss.

What do you mean by "consumer clouds"?

I refer to stuff like onedrive/gdrive/dropbox.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#779

Post by whatevaa »

alwa wrote: post #144
whatevaa wrote: post #121

Earlier quoted context omitted.

Yeah, I heard that consumer clouds are only locally redundant and there aren't even backups. So big DC damage could result in data loss.

I mean… at the risk of misinterpreting sarcasm— Except for the backup strategy said consumers apply to their data themselves, right? If I use a service called “it is stored in a datacenter in Virginia” then I will not be surprised when the meteor that hits Virginia destroys my data. For that reason I might also store copies of important things using the “it is stored in a datacenter in Oregon” service or something.

You might expect backups in case of fire, though. Even if data is not fully up to date.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#780

Post by majewsky »

throwaway2037 wrote: post #387
skissane wrote: post #286

Earlier quoted context omitted.

I spent a week of my life at a major insurance company in Seoul once, and the military style security, the obsession with corporate espionage, when all they were working on was an internal corporate portal for an insurance company… The developers had to use machines with no Internet access, I wasn’t allowed to bring my laptop with me lest I use it to steal their precious code. A South Korean colleague told me it was…

> South Korean corporate management is stuffed full of ex-military officers For those unaware, all "able-bodied" South Korean men are required to do about two years of military service. This sentence doesn't do much for me. Also, please remember that Germany also had required military service until quite recently. That means anyone "old" (over 40) and doing corp mgmt was probably also a military officer.

> That means anyone "old" (over 40) and doing corp mgmt was probably also a military officer.

Absolutely not. It was very common in Germany to deny military service and instead do a year of civil service as a replacement. Also, there were several exceptions from the """mandatory""" military service. I have two brothers who had served, so all I did was tick a checkbox and I was done with the topic of military service.

Post reply on HN