Wow. Maybe backups would have been a good idea.
Look at OVH a few years … they had backups in the same datacenter.
https://www.datacenterdynamics.com/en/news/ovhcloud-fire-rep...
Fire destroys S. Korean government's cloud storage system, no backups available
541–550 of 987 posts
Re: Fire destroys S. Korean government's cloud storage system, no backups available
#542Surely there must be something that's missing in translation? This feels like it simply can't be right.
I agree. No automated fire suppression system for critical infrastructure with no backup?
At first you think what an incompetent government would do such things, but even OVH pretty much did the same a few years ago. Destroyed some companies in the progress. A wooden floor in a datacenter with backups in the same building …
https://www.datacenterdynamics.com/en/news/ovhcloud-fire-rep...
Re: Fire destroys S. Korean government's cloud storage system, no backups available
#543Earlier quoted context omitted.
There are a lot of gray relations out there, but there’s almost no way you could morph the current US/SK relations to one of hostility; beyond a negligible minority of citizens in either being super vocal for some perceived slights.
You think when ICE arrested over 300 South Korean citizens who were setting up a Georgia Hyundai plant and subjected them to alleged human rights abuses, it was only a perceived slight? https://www.huffpost.com/entry/south-korea-human-rights-inve... How Trump’s ICE Raid Triggered Nationwide Outrage in South Korea https://www.newsweek.com/trump-ice-raid-hyundai-outrage-sout... 'The raid "will do lasting damage to Amer…
Yes.
Re: Fire destroys S. Korean government's cloud storage system, no backups available
#544Earlier quoted context omitted.
So right after the investigation was announced, they suddenly scheduled a UPS battery replacement which happened to start a fire big enough to destroy the entire data centre and all data or evidence? Yeah, that's way less suspicious, thanks for clearing that up.
UPS, check. Any kind of reasonable fire extinguisher, nah. A Kakao datacenter fire took the de-facto national chat app offline not too many years ago. Imagine operating a service that was nearly ubiquitous in the state of California and not being able to survive one datacenter outage. After reading the Phrack article, I don't know what to suspect, the typical IT disaster preparedness or the operators turning off the…
If the theory "north korea hacked the UPS batteries to blow" is true, though, then it makes more sense why fire suppression wasn't able to kick in on time.
Re: Fire destroys S. Korean government's cloud storage system, no backups available
#545While I am sure a huge portion of valuable work will be lost, I am smirking thinking of management making a call, "So, if there is any shadow IT who has been running mirror databases of valuable infrastructure, we would have a no questions asked policy on sharing that right now". I know that I have had to keep informal copies of valuable systems because the real source of truth is continually patched,offline,churn,wh…
Reminds me when Toy Story 2 was deleted and they found the backups on an artist's laptop that was working from home.
Re: Fire destroys S. Korean government's cloud storage system, no backups available
#546A lot of folks are arguing that the real problem is that they refused to use US cloud providers. No, that's not the issue. It's a perfectly reasonable choice to build your own storage infrastructure if it is needed. But the problem is they sacrificed "Availability" in pursuit of security and privacy. Losing your data to natural and man-made disasters is one of the biggest risks facing any storage infrastructure. Any…
Yeah, it's such a lame excuse to say "did not allow for external backups", as if that's a reasonable choice that they just couldn't work around.
South Korea isn't some poor backwater, they have tech companies and expertise, that they were "unable" to do backups was an intentional choice.
Re: Fire destroys S. Korean government's cloud storage system, no backups available
#547What structure could possibly preclude backups? I've never seen anything that couldn't be copied elsewhere. Maybe it was just convenient to have the possibility of losing everything.
I think that alluded to that earlier in the article: >However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost. I think they decided that their storage was too slow to allow backups? Seems hard to believe that they couldn't manage any backups... other sources said they had around 900TB of storage. An LTO-9 tape dri…
Let's run the numbers:
LTO-9 ~$92/tape in bulk. A 4 drive library with 80 drive capacity costs ~$40k* and can sustain about 1 Gbps. It also needs someone to barcode, inventory, and swap tapes once a week and an off-site vaulting provider like Iron Mountain. That's another $100k/year. Also, that tape library will need to be replaced every 4-7 years, so say 6 years. And those tapes wear out over X uses and sometimes go bad too. It might also require buying a server and/or backup/DR software too. Furthermore, a fire-rated data safe is recommended for about 1-2 weeks' worth of backups and spare media. Budget at least $200k/year for off-site tape backups for a minimal operation. (Let me tell you about the pains of self-destructing SSL2020 AIT-2 Sony drives.)
If backups for other critical services and this were combined, it would probably be cheaper to scale this kind of service rather reinventing the wheel for just one use-case in one department. That would allow for possibly multiple types of optimizations like network-based backups to nearline storage to then be streamed more directly to tape and using many more tape drives, possibly a tape silo robot(s) and perhaps split into 2-3 backup locations obviating the need for off-site vaulting.
Furthermore, it might be simpler, although more expensive, to operate another hot-/warm-site for backups and temporary business continuity restoration using a pile of HDDs and a network connection that's probably faster than that tape library. (Use backups, not replication because replication of errors to other sites is fail.)
Or the easiest option is to use one or more cloud vendors for even more $$$ (build vs. buy tradeoff).
* Traditionally (~20 years ago), enterprise "retail" prices of gear was sold at around 100% markup allowing for up to around 50% discount when negotiated in large orders. Enterprise gear also had a lifecycle of around 4.5 years while it still might technically work, there wouldn't be vendor support or replacements for them, and so enterprise customers are locked into perpetual planned obsolescence consumption cycles.
Re: Fire destroys S. Korean government's cloud storage system, no backups available
#548Goodness, I have over 100TB at home and it cost less than a two or three thousand dollars to put in place. That's like $25 per TB. > The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets. No, the 858TB amounts to under $25k for the government of the 10th largest economy, of one of the most sophisticated countries on the planet, to put in place. Two of those would be less than the price o…
You can buy a 24Tb drive on sale for $240 or so.
Sometimes I wonder why I still try and save disk space :-/
Re: Fire destroys S. Korean government's cloud storage system, no backups available
#549Earlier quoted context omitted.
I think that alluded to that earlier in the article: >However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost. I think they decided that their storage was too slow to allow backups? Seems hard to believe that they couldn't manage any backups... other sources said they had around 900TB of storage. An LTO-9 tape dri…
Let's run the numbers: LTO-9 ~$92/tape in bulk. A 4 drive library with 80 drive capacity costs ~$40k* and can sustain about 1 Gbps. It also needs someone to barcode, inventory, and swap tapes once a week and an off-site vaulting provider like Iron Mountain. That's another $100k/year. Also, that tape library will need to be replaced every 4-7 years, so say 6 years. And those tapes wear out over X uses and sometimes go…
$500K/year to back up a system used by 750,000 people is $0.66/year. Practically free.
At least now they see the true cost of not having any off site backups. It's a lot more than $0.66 per user.
Re: Fire destroys S. Korean government's cloud storage system, no backups available
#550Earlier quoted context omitted.
The article reads like they actually have a fault-tolerant system to store their data. This is probably a data dump for whatever files they are working with that might have started out as a cobbled-together prototype that just picked up momentum and pushed beyond its limitations. Many such cases not only in government IT...
Looking at the article, my read (which could be wrong) is that the backup was in the same room as the original.
No. It says that "most systems" in this data center are backed up to separate hardware on a different floor, and then a backup is made at a physically remote location.
This particular G-Drive system was not on the standard backup process - it sounds like it was much higher volume than any others, so maybe they couldn't use it. They did have a pilot going to get G-Drive backed up...it was supposed to be scaled up to the whole thing in December.