Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire destroys S. Korean government's cloud storage system, no backups available

koreajoongangdaily.joins.com

Post reply on HN
Print view View original

471–480 of 987 posts

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#471

Post by jiggawatts »

forinti wrote: post #168

What structure could possibly preclude backups? I've never seen anything that couldn't be copied elsewhere. Maybe it was just convenient to have the possibility of losing everything.

A key metric for recovery is the time it takes to read or write an entire drive (or drive array) in full. This is simply a function of the capacity and bandwidth, which has been getting worse and worse as drive capacities increase exponentially, but the throughput hasn't kept up at the same pace.

A typical 2005 era drive from two decades ago might have been 0.5 TB with a throughput of 70 MB/s for a full-drive transfer time (FDTT) of about 2 hours. A modern 32 TB drive is 64x bigger but has a throughput of only 270 MB/s which is less than 4x higher. Hence the FDDT is 33 hours!

This is the optimal scenario, things get worse in modern high-density disk arrays that may have 50 drives in a single enclosure with as little as 8-32 Gbps (1 GB/sec to 4 GB/sec) of effective bandwidth. That can push FDDT times out to many days or even weeks.

I've seen storage arrays where the drive trays were daisy chained, which meant that while the individual ports were fast, the bandwidth per drive would drop precipitously as capacity was expanded.

It's a very easy mistake to just keep buying more drives, plugging them in, and never going back to the whiteboard to rethink the HA/DR architecture and timings. The team doing this kind of BAU upgrade/maintenance is not the team that designed the thing originally!

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#472

Post by summerlight »

forinti wrote: post #168

What structure could possibly preclude backups? I've never seen anything that couldn't be copied elsewhere. Maybe it was just convenient to have the possibility of losing everything.

Basically it all boils down to budget. Those engineers knew this is a problem and wanted to fix that but that costs some money. And you know, bean counters in the treasury are basically like, "well it works well, why do we need that fix?" and the last conservative govt. was in a full spending cut mode. You know what happened there.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#473

Post by Sanzig »

solarengineer wrote: post #438
almostgotcaught wrote: post #396

Earlier quoted context omitted.

> My (rudimentary, layman) understanding is that encryption is almost like a last line of defense and should never be assumed to be unbreakable Lol this is woefully misinformed.

https://en.wikipedia.org/wiki/Post-quantum_cryptography It is my understanding that current encrypted content can someday be decrypted.

That's incorrect. Current asymmetric (ie: public-key) algorithms built using prime factoring or elliptic curve techniques are vulnerable to quantum attack using Shor's algorithm.

However, symmetric algorithms are not nearly as vulnerable. There is one known quantum attack using Grover's algorithm, but with quadratic speedup all it does is reduce the effective length of the key by half, so a 128-bit key will be equivalent to a 64-bit key and a 256-bit key will be equivalent to a 128-bit key. 256-bit keys are thus safe forever, since going down to a 128-bit key you are still talking age-of-the-universe break times. Even 128-bit keys will be safe for a very long time. While being reduced to a 64-bit key does make attacks theoretically possible, it is still tremendously difficult to do on a quantum computer, much harder than the asymmetric case (on the order of centuries even with very fast cycle times).

Finally, it's also worth noting that asymmetric cryptosystems are rapidly being updated to hybrid cryptosystems which add post-quantum algorithms (ie: algorithms which quantum computers are believed to provide little or no speedup advantage). So, going forward, asymmetric crypto should also no longer be vulnerable to store-now-decrypt-later attacks, provided there's no fundamental flaw in the new post-quantum algorithms (they seem solid, but they are new, so give the cryptographers a few years to try to poke holes in them).

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#474

Post by almostgotcaught »

solarengineer wrote: post #438
almostgotcaught wrote: post #396

Earlier quoted context omitted.

> My (rudimentary, layman) understanding is that encryption is almost like a last line of defense and should never be assumed to be unbreakable Lol this is woefully misinformed.

https://en.wikipedia.org/wiki/Post-quantum_cryptography It is my understanding that current encrypted content can someday be decrypted.

> someday be decrypted

Yup and that someday is the same day nuclear fusion is commercially viable.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#475

Post by RandomBacon »

sleepybrett wrote: post #381
rawgabbit wrote: post #367

Earlier quoted context omitted.

How large is this UPS that a fire can bring down all 96 servers? This story is really unbelievable.

depends on how many batteries were in the facility, if one goes up chances are the rest go too. Can halon systems not put out lithium fires?

I'm not sure about South Korea, but in the U.S., halon started to be phased out in 1994 due to its ozone-depleting characteristics. I believe new facilities use CO2.

I'm guessing lithium-ion batteries were not a factor years ago when those decisions were made.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#476

Post by xoa »

jacquesm wrote: post #347
xoa wrote: post #337

Earlier quoted context omitted.

> I'm aware of a big cloud services provider (I won't name any names but it was IBM) that lost a fairly large amount of data. Permanently. So that too isn't a guarantee. Permanently losing data at a given store point isn't relevant to losing data overall. Data store failures are assumed or else there'd be no point in backups. What matters is whether failures in multiple points happen at the same time , which means a…

> Permanently losing data at a given store point isn't relevant to losing data overall. I can't reveal any details but it was a lot more than just a given storage point. The interesting thing is that there were multiple points along the way where the damage would have been recoverable but their absolute incompetence made matters much worse to the point where there were no options left. > FWIW there's no "simply" abou…

>I can't reveal any details but it was a lot more than just a given storage point

Sorry, not brain not really clicking tonight and used lazy imprecise terminology here, been a long one. But what I meant by "store point" was any single data repository that can be interacted with as a unit, regardless of implementation details, that's part of a holistic data storage strategy. So in this case the entirety of IBM would be a "storage point", and then your own self-hosted system would be another, and if you also had data replicated to AWS etc those would be others. IBM (or any other cloud storage provider operating in this role) effectively might as well simply be another hard drive. A very big, complex and pricey magic hard drive that can scale its own storage and performance on demand granted, but still a "hard drive".

And hard drives fail, and that's ok. Regardless of the internal details of how the IBM-HDD ended up failing, the only way it'd affect the overall data is if that failure happened simultaneously with enough other failures at local-HDD and AWD-HDD and rsync.net-HDD and GC-HDD etc etc that it exceeded available parity to rebuild. If these are all mirrors, then only simultaneous failure of every single last one of them would do it. It's fine for every single last one of them to fail... just separately, with enough of a time delta between each one that the data can be rebuilt on another.

>If you can't do the job you should get out of the kitchen.

Isn't that precisely what bringing in external entities as part of your infrastructure strategy is? You're not cooking in their kitchen.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#477

Post by refulgentis »

AnotherGoodName wrote: post #405
throwaway2037 wrote: post #372

At the very bottom of the article, I see this notice: > This article was originally written in Korean and translated by a bilingual reporter with the help of generative AI tools. It was then edited by a native English-speaking editor. All AI-assisted translations are reviewed and refined by our newsroom. I like that. It is direct and honest. I'm fine with people using LLMs for natural language related work, as long a…

Especially since LLM tech was originally developed for translation. That’s the original reason so much work was done to create a model that could handle context and it turned out that was helpful in more areas than just translation. While LLM usage is just spinning up in other areas, for translation they have been doing this job well for over 5 years now.

> While LLM usage is just spinning up in other areas,

Oh?

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#478

Post by fragmede »

solarengineer wrote: post #438
almostgotcaught wrote: post #396

Earlier quoted context omitted.

> My (rudimentary, layman) understanding is that encryption is almost like a last line of defense and should never be assumed to be unbreakable Lol this is woefully misinformed.

https://en.wikipedia.org/wiki/Post-quantum_cryptography It is my understanding that current encrypted content can someday be decrypted.

Someday, theoretically, maybe. This means that, as far as everyone knows, if I properly secure a message to you using RSA, no one else is reading the message. Maybe in 50 years they can, but, well, that's in 50 years. Alarmists would have you believe it'll happen in three. I'm just an Internet rando, but my money's on it being closer to 50. Regardless though, it's not today.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#479

Post by ateng »

chatmasta wrote: post #452
StopDisinfo910 wrote: post #240

Earlier quoted context omitted.

The issue here is not refusing to use a foreign third party. That makes sense. The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk. That’s gross mismanagement.

Nothing increases the risk of servers catching fire like government investigators showing up to investigate allegations that North Korea hacked the servers.

It _almost_ sounds like you're suggesting the fire was deliberate!

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#480

Post by aspenmayer »

Diti wrote: post #469
hackernewds wrote: post #458

Earlier quoted context omitted.

why would you not be fine about it?

You probably don’t want to read news websites which are nothing but LLM output without a journalist reviewing the articles. Unless you’re a fan of conspiracy theories or ultra-aligned content.

Case in point:

A New Gaza Rage Machine–With Polish Origins - https://news.ycombinator.com/item?id=45453533

Post reply on HN