Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire destroys S. Korean government's cloud storage system, no backups available

koreajoongangdaily.joins.com

Post reply on HN
Print view View original

801–810 of 987 posts

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#801

Post by kube-system »

bombcar wrote: post #311
eCa wrote: post #120

Earlier quoted context omitted.

Agree completely that it's absolute wild to run such a system without backups. But at this point no government should keep critical data on foreign cloud storage.

If you can’t encrypt your backups such that you could store them tatooed on Putin’s ass, you need to learn about backups more.

Governments need to worry about

1. future cryptography attacks that do not exist today

2. Availability of data

3. The legal environment of the data

Encryption is not a panacea that solves every problem

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#803

Post by DoctorOetker »

indolering wrote: post #676
yetihehe wrote: post #623

Earlier quoted context omitted.

There is a reason that NIST recommends new encryption algorithms from time to time. If you get a copy of ALL government data, in 20 years you might be able to break encryption and get access to ALL government data from 20yr ago, no matter how classified they were, if they were stored in that cloud. Such data might still be valuable, because not all data is published after some period.

That's going away. We are seeing reduced deprecations of crypto algorithms over time AFAICT. The mathematical foundations are becoming better understood and the implementations' assurance levels are improving too. I think we are going up the bathtub curb here. The value of said data diminishes with time too. You can totally do an off-site cloud backup with mitigation fallbacks should another country become unfriendly…

I don't consider myself special, anything I can find eventually proof assistants using ML will find...

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#804

Post by namibj »

hnlmorg wrote: post #743
cm2187 wrote: post #733

Earlier quoted context omitted.

Plus long distance was mostly fibre already. And even regular electrical wires aren’t really much slower than fibre in term of latency. Parent probably meant bandwidth.

Copper doesn't work over these kinds of distances without powered switches, which adds latency. And laying fibre over several miles would be massively expensive. Well outside the realm of all but the largest of corporations. There's a reason buildings with high bandwidth constraints huddle near internet backbones. What used to happen (and still does as far as I know, but I've been out of the networking game for a whi…

> There's a reason buildings with high bandwidth constraints huddle near internet backbones.

Yeah because interaction latency matters and legacy/already buried fiber is expensive to rent so you might as well put the facility in range of (not-yet-expensive) 20km optics.

> Copper doesn't work over these kinds of distances without powered switches, which adds latency.

You need a retimer, which adds on the order of 5~20 bits of latency.

> And that becomes a huge problem if you want to do any real-time work like DB replicas.

Almost no application would actually require "zero lost data", so you could get away with streaming a WAL or other form of reliably-replayable transaction log and cap it to an acceptable number of milliseconds of data loss window before applying blocking back pressure. Usually it'd be easy to tolerate enough for the around 3 RTTs you'd really want to keep to cover all usual packet loss without triggering back pressure.

Sure, such a setup isn't cheap, but it's (for a long while now) cheaper than manually fixing the data from the day your primary burned down.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#805

Post by someuser2345 »

StopDisinfo910 wrote: post #240
kristianc wrote: post #95

The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for extern…

The issue here is not refusing to use a foreign third party. That makes sense. The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk. That’s gross mismanagement.

> The issue here is not refusing to use a foreign third party. That makes sense.

For anyone else who's confused, G-Drive means Government Drive, not Google Drive.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#806

Post by StopDisinfo910 »

hnlmorg wrote: post #743
cm2187 wrote: post #733

Earlier quoted context omitted.

Plus long distance was mostly fibre already. And even regular electrical wires aren’t really much slower than fibre in term of latency. Parent probably meant bandwidth.

Copper doesn't work over these kinds of distances without powered switches, which adds latency. And laying fibre over several miles would be massively expensive. Well outside the realm of all but the largest of corporations. There's a reason buildings with high bandwidth constraints huddle near internet backbones. What used to happen (and still does as far as I know, but I've been out of the networking game for a whi…

To be fair, tape backups are very much ok as a disaster recovery solution. It's cheap once you have the tape drive. Bandwith is mostly fine if you want to read them sequentially. It's easy to store and handle and fairly resistant.

It's "only" poor if you need to restore some files in the middle or want your backup to act as a failover solution to minimise unavailability. But as a last resort solution in case of total destruction, it's pretty much unbeatable cost-wise.

G-Drive was apparently storing less than 1PB of data. That's less than 100 tapes. I guess some files were fairly stable so completely manageable with a dozen of tape drives, delta storage and proper rotation. We are talking of a budget of what 50k$ to 100k$. That's peanuts for a project of this size. Plus the tech has existed for ages and I guess you can find plenty of former data center employees with experience handling this kind of setup. They really have no excuse.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#807

Post by ZiiS »

Are we actually sure they didn't do due diligence?

This is the individual's work files of civil servants. These will overwhelmingly be temporary documents they were legally obliged to delete at some point in the last 8 years. Any official filings or communications would have been to systems of record that were not effected.

This is more that a very large fire, probably unlucky for once a decade, caused civil servants to lose hours of work in files they were working on. A perfect system could have obviously prevented this and ensured availability, but not without cost.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#808

Post by marcusb »

stego-tech wrote: post #773
StopDisinfo910 wrote: post #240

Earlier quoted context omitted.

The issue here is not refusing to use a foreign third party. That makes sense. The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk. That’s gross mismanagement.

This. Speaking specifically from the IT side of things, an employer or customer refusing to do backups is the biggest red flag I can get, an immediate warning to run the fuck away before you get blamed for their failure, stego-tech kind of situation. That being said, I can likely guess where this ends up going: * Current IT staff and management are almost certainly scapegoated for “allowing this to happen”, despite t…

> * Everyone involved is unlikely to find work again anytime soon once names are bandied about in investigations

They might (MIGHT) get fired from their government jobs, but I'll bet they land in consulting shops because of their knowledge of how the government's IT teams operate.

I'll also bet the internal audit team slides out of this completely unscathed.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#809

Post by rtkwe »

Well it's really in the cloud(s) now! /s

No offsite backups is a real sin, sounds like a classic case where the money controllers thought 'cloud' automatically meant AWS level redundant cloud and instead they had a fancy centralized datacenter with insufficient backups.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#810

Post by disgruntledphd2 »

marcusb wrote: post #808
stego-tech wrote: post #773

Earlier quoted context omitted.

This. Speaking specifically from the IT side of things, an employer or customer refusing to do backups is the biggest red flag I can get, an immediate warning to run the fuck away before you get blamed for their failure, stego-tech kind of situation. That being said, I can likely guess where this ends up going: * Current IT staff and management are almost certainly scapegoated for “allowing this to happen”, despite t…

> * Everyone involved is unlikely to find work again anytime soon once names are bandied about in investigations They might (MIGHT) get fired from their government jobs, but I'll bet they land in consulting shops because of their knowledge of how the government's IT teams operate. I'll also bet the internal audit team slides out of this completely unscathed.

> I'll also bet the internal audit team slides out of this completely unscathed.

They really, really shouldn't. However, if they were shouted down by management (an unfortunately common experience) then it's on management.

The trouble is that you can either be effective at internal audit or popular, and lots of CAE's choose the wrong option (but then, people like having jobs so I dunno).

Post reply on HN