Live data from Hacker News

Fire destroys S. Korean government's cloud storage system, no backups available

koreajoongangdaily.joins.com

511–520 of 987 posts

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#511

Earlier quoted context omitted.

They are overwhelmingly whitelabeled providers. For example, Samsung SDI Cloud (the largest "Korean" cloud) is an AWS white label. Korea is great at a lot of engineering disciplines. Sadly, software is not one of them, though it's slowly changing. There was a similar issue a couple years ago where the government's internal intranet was down a couple days because someone deployed a switch in front of outbound connecti…

I am very happy with the software that powers my Hyundai Tuscon hybrid. (It's a massive system that runs the gas and electric engines, recharging, shifting gears, braking, object detection, and a host of information and entertainment systems.) After 2 years, 0 crashes and no observable errors. Of course, nothing is perfect: maps suck. The navigation is fine; it's the display that is at least 2 decades behind the time…

I've been working for a Korean Hyundai supplier for two years training them in modern software development processes. The programming part is not a problem, they have a lot of talented people.

The big problem from my point of view is management. Everyone pushes responsibility and work all the way down to the developera so that they do basically everything themselves from negotiating with the customer, writing the requirements (or not) to designing the architecture, writing the code and testing the system.

If they're late,they just stay and work longer and on the weekends and sleep at the desk.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#512
post #473

Earlier quoted context omitted.

https://en.wikipedia.org/wiki/Post-quantum_cryptography It is my understanding that current encrypted content can someday be decrypted.

That's incorrect. Current asymmetric (ie: public-key) algorithms built using prime factoring or elliptic curve techniques are vulnerable to quantum attack using Shor's algorithm. However, symmetric algorithms are not nearly as vulnerable. There is one known quantum attack using Grover's algorithm, but with quadratic speedup all it does is reduce the effective length of the key by half, so a 128-bit key will be equiva…

>However, symmetric algorithms are not nearly as vulnerable. There is one known quantum attack using Grover's algorithm, but with quadratic speedup all it does is reduce the effective length of the key by half, so a 128-bit key will be equivalent to a 64-bit key and a 256-bit key will be equivalent to a 128-bit key. 256-bit keys are thus safe forever, since going down to a 128-bit key you are still talking age-of-the-universe break times. Even 128-bit keys will be safe for a very long time. While being reduced to a 64-bit key does make attacks theoretically possible, it is still tremendously difficult to do on a quantum computer, much harder than the asymmetric case (on the order of centuries even with very fast cycle times).

Specifically it's worth noting here the context of this thread: single entity data storage is the textbook ideal case for symmetric. While Shor's "only" applies [0] to one type of cryptography, that type is the key to the economic majority of what encryption is used for (the entire world wide web etc). So it still matters a lot. But when you're encrypting your own data purely to use it for yourself at a future time, which is the case for your own personal data storage, pure symmetric cryptography is all you need (and faster). You don't have the difficult problem of key distribution and authentication with the rest of humanity at all and can just set that aside entirely. So to the point of "why not back up data to multiple providers" that "should" be no problem if it's encrypted before departing your own trusted systems.

Granted, the "should" does encompass some complications, but not in the math or software, rather in messier aspects of key control and metadata. Like, I think an argument could be made that it's easier to steal a key then exfiltrate huge amounts of data without someone noticing, but there's powerful enough tools for physically secure key management (and splitting, Shamir's Secret Sharing means you can divide up each unique service backup encryption key into an arbitrary number of units and then require an arbitrary number of them to all agree to reconstitute the usable original key) that I'd expect an advanced government to be able to handle it, more so then data at rest even. Another argument is that even if a 3rd party cannot ever see anything about the content of an encrypted archive, they can get some metadata from its raw size and the flows in and out of it. But in the reduced single use case of pure backups where use is regular widely spaced dumps, and for something as massive as an entire government data cloud with tens of thousands of uncorrelated users, the leakage of anything meaningful seems low. And of course both have to be weighed against a disaster like this one.

Anyway, all well above my pay grade. But if I were a citizen there I'd certainly be asking questions because this feels more like NIH or the political factors influencing things.

----

0: Last I checked there was still some serious people debating on whether it will actually work out in the real world, but from the perspective of considering security risk it makes sense to just take it as given that it will work completely IRL, including that general purpose quantum computers that can run it will prove sufficiently scalable to provide all the needed qubits.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#513

Earlier quoted context omitted.

A year ago, I would have easily claimed the same thing about Denmark.

I don't follow. Can you share more context?

The US is threatening to invade Greenland, what means active war with Denmark.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#514
post #186
post #127

Earlier quoted context omitted.

Why not? If the region is in country, encrypted, and with proven security attestations validated by third parties, a backup to a cloud storage would be incredibly wise. Otherwise we might end up reading an article about a fire burning down a single data center

Microsoft has already testified that the American government maintains access to their data centres, in all regions. It likely applies to all American cloud companies. America is not a stable ally, and has a history of spying on friends. So unless the whole of your backup is encrypted offline, and you trust the NSA to never break the encryption you chose, its a national security risk.

There are no stable allies. No country spies on its friends because countries don't have friends, they have allies. And everybody spies on their allies.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#515
post #341

Earlier quoted context omitted.

> The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets. This attempt at putting it in perspective makes me wonder what would put it in perspective. "100M sets of harry potter novels" would be one step in the right direction, but nobody can imagine 100M of anything either. Something like "a million movies" wouldn't work because they are very different from text media in terms of how much…

"equivalent to 50 hard drives" ?

I don't think many people have 16TB of storage on the hard drives they're familiar with but I take your point nonetheless! Simple solution. I was apparently too focused on the information factor and didn't think of just saying how many of some storage medium it is equivalent to, which makes a lot of sense indeed

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#516
post #476

Earlier quoted context omitted.

> Permanently losing data at a given store point isn't relevant to losing data overall. I can't reveal any details but it was a lot more than just a given storage point. The interesting thing is that there were multiple points along the way where the damage would have been recoverable but their absolute incompetence made matters much worse to the point where there were no options left. > FWIW there's no "simply" abou…

> I can't reveal any details but it was a lot more than just a given storage point Sorry, not brain not really clicking tonight and used lazy imprecise terminology here, been a long one. But what I meant by "store point" was any single data repository that can be interacted with as a unit, regardless of implementation details, that's part of a holistic data storage strategy. So in this case the entirety of IBM would…

Ah ok, clear. Thank you for the clarification. Some more interesting details: the initial fault was triggered by a test of a fire suppression system, that would have been recoverable. But someone thought they were exceedingly clever and they were going to fix this without any downtime and that's when a small problem became a much larger one, more so when they found out that their backups were incomplete. I still wonder if they ever did RCA/PM on this and what their lessons learned were. It should be a book sized document given how much went wrong. I got the call after their own efforts failed by one of their customers and after hearing them out I figured this is not worth my time because it just isn't going to work.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#517
post #404

Earlier quoted context omitted.

[flagged]

As a sysadmin at company that provide fairly sensitive services, I find online cloud backups to be way to slow for the purpose of protecting against something like the server room being destroyed by a fire. Even something like spinning disks at a remote location feel like a risk, as files would need to be copied onto faster disks before services could be restored, and that copying would take precious time during an e…

SK would be totally fine with that though because that means there would eventually be recovery!

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#518

after the kakao fire incident and now this i struggle to understand how they got so advanced in other areas. this is like amateur hour level shit.

It is the same in Japan. They are really good for hardware and other "physical" engineering disciplines, but they are terrible when it comes to software and general IT stuff. Seriously, I work here as an IT guy and I can't stop wondering how they could become so advance in other areas and stay so backwards in anything software-related except videogames.

Yeah. This is my exact experience too wrt japan! The japanese just somehow can't assess and manage neither the scale, nor the complexity, the risk, the effort or the cost of software projects. Working in japan as a software guy feels like working in a country lagging 30-40 years behind :/

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#519
post #509
post #500

Earlier quoted context omitted.

The most sophisticated countries and companies are smart enough to use the least sophisticated backup methods. SK needs to backup their data to cassette tapes and tape libraries cost a bit more than that, but not much. Even if they boat their tapes over to an iron mountain in the US, I can't imagine the equipment and service fees are going to cost them more than a few hundred grand. They'll be spending more on the he…

The operational expenses of this stuff dwarfs the hardware cost. For the tape mountain, you need robots to confirm the tapes still work (mean time to detection of device failure and recovery are key for RAID durability computations). So, someone needs to constantly repair the robots or whatever. If I was being paid to manage that data set, I’d probably find two enterprise storage vendors, and stick two copies of the…

even with dual vendors, you'd have to still put in place a backup/restore procedures (with the associated software, which may need to be custom). Then you'd need regular testing. These operational concerns will basically double the cost yearly, probably.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#520

Earlier quoted context omitted.

In Korea, shame often serves as the primary motivator behind high-profile suicides. It's rooted in the cultural concept of "chemyeon (체면)", which imposes immense pressure to maintain a dignified public image.

Do you have any example of these high profile suicides that can't be better explained as "taking one for the team" for lack of a better idiom. Shame is a powerful social force throughout the society, but we're talking about systematic screwings more often than not backed by political corruption (letting incompetent entities deal with gov contract on basis of political money and other favors) or straight fraud.

[deleted]
Post reply on HN