Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire destroys S. Korean government's cloud storage system, no backups available

koreajoongangdaily.joins.com

Post reply on HN
Print view View original

691–700 of 987 posts

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#691

Post by hnlmorg »

flumpcakes wrote: post #673
AdamN wrote: post #670

Earlier quoted context omitted.

Jersey City still was fine and 50 miles can be problematic for certain types of backup (failover) protocols. Regular tape backups would be fine but secondary databases can't be that far away (at least not at the time). I remember my boss at WFC saying that the most traffic over the data lines was in the middle of the night due to backups - not when everybody was in the office.

Companies big enough will lay the fibre. 50-100 miles of fibre isn't much if you are a billion dollar business. Even companies like BlackRock who had their own datacenters have since taken up Azure. 50 miles latency is negligible, even for databases.

The WTC attacks were in the 90s and early 00s and back then, 50 miles of latency was anything but negligible and Azure didn’t exist.

I know this because I was working on online systems back then.

I also vividly remember 9/11 and the days that followed. We had a satellite dish with multiple receivers (which wasn’t common back then) so had to run a 3rd party Linux box to descramble the single. We watch 24/7 global news on a crappy 5:4 CRT running Windows ME during the attack. Even in the UK, it was a somber and sobering experience.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#692

Post by rstuart4133 »

admiralrohan wrote: post #655

Is there any solution to these kind of issues other than having multiple backups and praying that not all of them will caught fire at the same time?

Backups are best thought of as a multi dimensional problem, as in, they can be connected in many dimensions. Destroy a backup, and all those in the same dimension are also destroyed. This means you must have to have redundancy in many dimensions. That all sounds a bit abstract, so ...

One dimension is two backups can be close in space (ie, physically close, as happened here). Ergo backups must be physically separated.

You've heard RAID can't be a backup? Well it sort of can, and the two drives can be physically separated in space. But they are connected in another dimension - time, as in they reflect the data at the same instant in time. So if you have a software failure that corrupts all copies, your backups are toast as you can't go back to a previous point in time to recover.

Another dimension is administrative control. Google Drive for example will backup your stuff, and separate it in space and time. But they are connected by who controls them. If you don't pay the bill or piss Google off, you've lost all your backups. I swear every week I see a headline saying someone lost their data this way.

Then backups can be all connected to you via one internet link, or connected to one electrical grid, or even one country that goes rogue. All of those are what I called dimensions, that you have to ensure your backups are held at a different location in each dimension.

Sorry, that didn't answer your question. The answer no. It's always possible all copies could be wiped out at the same time. You are always relying on luck, and perhaps prayer if you think that helps your luck.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#693

Post by j45 »

kristianc wrote: post #95

The government official who insisted that commercial AWS/GCP/Azure couldn't possibly be trusted with keeping the information will be keeping their head low for a few days then... "The Interior Ministry explained that while most systems at the Daejeon data center are backed up daily to separate equipment within the same center and to a physically remote backup facility, the G-Drive’s structure did not allow for extern…

They put everything only in one datacenter. A datacenter located elsewhere should have been setup to mirror.

This has nothing to do with commercial clouds. Commercial clouds are just datacenters. They could pick one commercial cloud data center and not do much more to mirror or backup in different regions. I understand some of the services have inherent backups.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#694

Post by myrion »

egorfine wrote: post #669
leipert wrote: post #666

Earlier quoted context omitted.

Nope: The other way around. If you are of a certain size, you are required to ensure certain criteria. NIS-2 is the EU directive and it more or less maps to ISO27001 which includes risk management against physical catastrophes. https://www.openkritis.de/eu/eu-nis-2-germany.html Of course you can do backups if you are smaller, or comply with such a standard if you so wish.

[flagged]

Considering that companies will do everything to avoid doing sensible things that cost money - yes, of course the government has to step in and mandate things like this.

It's no different from safety standards for car manufacturers. Do you think it's ridiculous that the government tells them how to build cars?

And similarly here: If the company is big enough / important enough, then the cost to society if their IT is all fucked up is big enough that the government is justified in ensuring minimum standards. Including for backups.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#695

Post by 9dev »

JumpCrisscross wrote: post #272
9dev wrote: post #246

Earlier quoted context omitted.

That doesn’t capture the full truth. Since Snowden, we have hard evidence the NSA has been snooping on foreign governments and citizens alike with the purpose of harvesting data and gathering intelligence, not just to verify their loyalty. No nation should trust the USA, especially not with their state secrets, if they can help it. Not that other countries are inherently more trustworthy, but the US is a known bad ac…

> Since Snowden, we have hard evidence the NSA has been snooping on foreign governments and citizens alike We also know this is also true for Russia, China and India. Being spied on is part of the cost of relying on external security guarantees. > Not that other countries are inherently more trustworthy, but the US is a known bad actor All regional and global powers are known bad actors. That said, Seoul is already i…

> All regional and global powers are known bad actors.

That they are. Americans tend to view themselves as "the good guys" however, which is a wrong observation and thus needs pointing out in particular.

> That said, Seoul is already in bed with Washington. Sending encrypted back-ups to an American company probably doesn't increase its threat cross section materially.

If they have any secrets they attempt to keep even from Washington, they are contained in these backups. If that is the case, storing them (even encrypted) with an American company absolutely compromises security, even if there is no known threat vector at this time. The moment you give up control of your data, it will forever be subject to new threats discovered afterward. And that may just be something like observing the data volume after an event occurs that might give something away.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#696

Post by egorfine »

hiharryhere wrote: post #684
egorfine wrote: post #669

Earlier quoted context omitted.

[flagged]

Is it? It would be incredible if the government didn’t have specific requirements for critical infrastructure. Say you’re an energy company and an incident could mean that a big part of the country is without power, or you’re a large bank and you can’t process payroll for millions of workers. They’re ability to recover quickly and completely matters. Just recently in Australia an incident at Optus, a large phone comp…

[flagged]

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#697

Post by egorfine »

hiharryhere wrote: post #687
egorfine wrote: post #682

Earlier quoted context omitted.

> it absolutely does make sense for the government to force such companies Problem is, a) governments are infiltrated by russian assets and b) governments are known to enforce detrimental IT regulations. Germany especially so. > power plants, telco infra, traffic infrastructure or hospitals Their system _will_ get hit by ransomware or APTs. It is not possible to mandate common sense or proper IT practices, no matter…

Government isn’t perfect but I’d be interested to know what alternative you propose?

a) Incarceration time for IT execs and responsible engineers.

b) Let companies go out of business once they fail to protect their own crucial data.

None of that is possible.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#698

Post by 9dev »

signatoremo wrote: post #365
9dev wrote: post #246

Earlier quoted context omitted.

That doesn’t capture the full truth. Since Snowden, we have hard evidence the NSA has been snooping on foreign governments and citizens alike with the purpose of harvesting data and gathering intelligence, not just to verify their loyalty. No nation should trust the USA, especially not with their state secrets, if they can help it. Not that other countries are inherently more trustworthy, but the US is a known bad ac…

There is no such thing as good or trustworthy actors when it comes to state affairs. Each and every one attempt to spy on the others. Perhaps US have more resources to do so than some others. You really have no evidence to back up your assertion, because you’d have to be an insider.

> There is no such thing as good or trustworthy actors when it comes to state affairs. Each and every one attempt to spy on the others. Perhaps US have more resources to do so than some others.

Perhaps is doing a lot of work here. They do, and they are. That is what the Snowden leaks proved.

> You really have no evidence to back up your assertion, because you’d have to be an insider.

I don't, because the possibility alone warrants the additional caution.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#699

Post by StopDisinfo910 »

egorfine wrote: post #669
leipert wrote: post #666

Earlier quoted context omitted.

Nope: The other way around. If you are of a certain size, you are required to ensure certain criteria. NIS-2 is the EU directive and it more or less maps to ISO27001 which includes risk management against physical catastrophes. https://www.openkritis.de/eu/eu-nis-2-germany.html Of course you can do backups if you are smaller, or comply with such a standard if you so wish.

[flagged]

It’s government telling you the minimum you have to do. There is nothing incredible there.

It makes sense that as economic operators become bigger, as the impact of their potential failure grows on the rest of the economy, they have to become more resilient.

That’s just the state forcing companies to take externalities into account which is the state playing its role.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#700

Post by osivertsson »

hnlmorg wrote: post #691
flumpcakes wrote: post #673

Earlier quoted context omitted.

Companies big enough will lay the fibre. 50-100 miles of fibre isn't much if you are a billion dollar business. Even companies like BlackRock who had their own datacenters have since taken up Azure. 50 miles latency is negligible, even for databases.

The WTC attacks were in the 90s and early 00s and back then, 50 miles of latency was anything but negligible and Azure didn’t exist. I know this because I was working on online systems back then. I also vividly remember 9/11 and the days that followed. We had a satellite dish with multiple receivers (which wasn’t common back then) so had to run a 3rd party Linux box to descramble the single. We watch 24/7 global news…

Laws of physics hasn't changed since the early 00s though, we could build very low latency point to point links back then too.
Post reply on HN