Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fire destroys S. Korean government's cloud storage system, no backups available

koreajoongangdaily.joins.com

Post reply on HN
Print view View original

571–580 of 987 posts

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#571

Post by tra3 »

southernplaces7 wrote: post #564
covercash wrote: post #302

Earlier quoted context omitted.

If the US government and corporate executives had even half this level of shame, we'd have nobody left in those positions!

Not the same country but another example of a culturally similar attitude towards shame over failure: In Japan in 1985, Flight 123, a massive Boeing 747 carrying 524 people, lost control shortly after takeoff from Tokyo en route to Osaka. The plane's aft pressure bulkhead catastrophically exploded, causing total decompression at the high altitude, severing all four of the massive plane's hydraulic stabilizer systems…

What an incredible story. Thanks for sharing.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#572

Post by bedstefar »

DaiPlusPlus wrote: post #567
WalterBright wrote: post #548

Earlier quoted context omitted.

You can buy a 24Tb drive on sale for $240 or so. Sometimes I wonder why I still try and save disk space :-/

Link? Am both curious and skeptical

Not trying to account for the parent's claim, but generally check diskprices.com for the latest deals on Amazon (.com, .co.uk, .de, .es, .it etc)

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#573

Post by vitorgrs »

acchow wrote: post #536
StopDisinfo910 wrote: post #240

Earlier quoted context omitted.

The issue here is not refusing to use a foreign third party. That makes sense. The issue is mandating the use of remote storage and not backing it up. That’s insane. It’s like the most basic amount of preparation you do. It’s recommended to even the smallest of companies specifically because a fire is a risk. That’s gross mismanagement.

> The issue here is not refusing to use a foreign third party. That makes sense. Encrypt before sending to a third party?

Would you think that the U.S would encrypt gov data and store on Alibaba's Cloud? :)

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#574

Post by jeroenhd »

yongjik wrote: post #13
benoau wrote: post #2

> However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost. Yikes. You'd think they would at least have one redundant copy of it all. > erasing work files saved individually by some 750,000 civil servants > 30 gigabytes of storage per person That's 22,500 terabytes, about 50 Backblaze storage pods. Or even just mir…

It's even worse. According to other articles [1], the total data of "G drive" was 858 TB. It's almost farcical to calculate, but AWS S3 has pricing of about $0.023/GB/month, which means the South Korean government could have reliable multi-storage backup of the whole data at about $20k/month. Or about $900/month if they opted for "Glacier deep archive" tier ($0.00099/GB/month). They did have backup of the data ... in…

AWS? Linus Tech Tips has run multiple petabyte servers in their server closet just for sponsor money and for the cool of it. No need to outsource your national infrastructure to foreign governments, a moderate (in government terms) investment in a few racks across the country could've replicated everything for maybe half a year's worth of Amazon subscription fees.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#575

Post by __turbobrew__ »

comprev wrote: post #66
zaphar wrote: post #10

Earlier quoted context omitted.

Ironically many of those documents for procedures probably lived on that drive...

You jest, but I once had a client who's IaC provisioning code was - you guessed it - stored on the very infrastructure which got destroyed.

If you are one of the big boys (FAANG and other large companies who run physical infra) you will have this problem as well. The infra systems run and replace themselves and if something fundamental breaks (for example, your deployment system requires DNS, but your DNS servers are broken, but you cannot deploy to fix them as the deploy service requires DNS).

From what I have seen a lot of time the playbooks to fix these issues are just rawdogging files using rsync manually. Ideally you deploy your infrastructure in cells where rollouts proceed cell by cell so you can catch issues sooner and also implement failover to bootstrap broken cells (in my DNS example, client could talk to DNS servers in the closest non-broken cell using BGP based routing). It is hard to test, and there are some global services (like that big Google outage a few months ago was due to the global auth service being down).

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#576

Post by jeroenhd »

baobabKoodaa wrote: post #102
benoau wrote: post #2

> However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained — meaning all data has been permanently lost. Yikes. You'd think they would at least have one redundant copy of it all. > erasing work files saved individually by some 750,000 civil servants > 30 gigabytes of storage per person That's 22,500 terabytes, about 50 Backblaze storage pods. Or even just mir…

You're assuming average worker utilized the full 30G of storage. More likely average was at like 0.3G.

On the other hand: backups should also include a version history of some kind, or you'd be vulnerable to ransomware.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#577

Post by [deleted] »

senko wrote: post #338
vbezhenar wrote: post #313

Earlier quoted context omitted.

It is much more likely and cheaper, that US marines will desant and capture your backup facility, than someone would break AES-128.

Sending troops would be an act of war, and definitely not cheap. Stealing some encryption keys, just another Wednesday.

[deleted]

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#578

Post by awesomeusername »

I'm CTO of a TINY company, with pretty much exactly half this data. I run all storage and offside backups personally, because I can't afford a full time sysadmin yet.

And the cost of everything is PAIN to us.

If our building burned down we would lose data, but only the data we are Ok with losing in a fire.

I'd love to know the real reason. It's not some useless tech... it's politics, surely.

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#579

Post by import »

DaiPlusPlus wrote: post #567
WalterBright wrote: post #548

Earlier quoted context omitted.

You can buy a 24Tb drive on sale for $240 or so. Sometimes I wonder why I still try and save disk space :-/

Link? Am both curious and skeptical

Very possible with sales and especially non nas grade harddisks

Re: Fire destroys S. Korean government's cloud storage system, no backups available

#580

Post by eredengrin »

sexy_seedbox wrote: post #388
Frost1x wrote: post #137

Earlier quoted context omitted.

Having just visited South Korea last year, one thing that sort of caught me off guard was the lack of Google Maps or other major direction system. I wasn’t aware but turns out anything considered “detailed mapping” infrastructure has to be ran stored and on South Korean soil, probably lots of other requirements. So you’re stuck with some shotty local mapping systems that are just bad. There may be a point in time it…

Why didn't you use Kakao Maps or Naver Maps? They're not shotty and work just fine, even if you don't read Korean, you can quickly guess the UI based on the icons.

Agree, Naver maps for navigating public transit in Seoul is excellent. Easier to figure out than public transit in any American city I've been to and I don't read or speak Korean. iirc it even tells the fastest routes/best carriage to be on to optimize transferring between lines.
Post reply on HN