Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google loses data as lightning strikes

bbc.com

Post reply on HN
Print view View original

121–130 of 142 posts

Re: Google loses data as lightning strikes

#121

Post by mikeash »

Dylan16807 wrote: post #111
mikeash wrote: post #77

Earlier quoted context omitted.

Do they spell out what assumptions they're working under? Because, for example, I'm pretty sure that the odds of a civilization-ending asteroid or comet strike in the next year, while quite low, are higher than what's implied by 99.999999999% durability on trillions of objects."

Why do you specify trillions of objects? They practically promise data loss at that scale. Also, the typical civilization-ending comet or asteroid is going to destroy 0 or 1 geographically-diverse data centers. Remember that this is a retention promise, not an uptime promise.

Just because it's how many objects they store. But now that you mention it, it has no relevance to the probability of data loss due to global catastrophe, since it cancels out.

If you're proposed reliability is data loss of one in 100 billion per year, that assumes the risk of e.g. global thermonuclear war is no more than that amount, or you're ignoring it for the purposes of the calculation. Which is why I wonder about their assumptions.

As for a comet destroying at most one data center, if it really ends civilization then the data will probably be lost before too much longer even if the facility physically survives.

Re: Google loses data as lightning strikes

#122

Post by megablast »

pdabbadabba wrote: post #109
Tloewald wrote: post #68

Earlier quoted context omitted.

So empirically speaking their reliability is far, far worse than stated just based on that one data point.

Who knows. Maybe his is the last data Glacier will lose in the next [suitably large number] years. That's bad luck, buddy!

I too choose to extrapolate all my losses over the next 100 billion years.

Re: Google loses data as lightning strikes

#123

Post by wmf »

pacala wrote: post #116
wmf wrote: post #79

Earlier quoted context omitted.

Not for block storage (aka persistent disks): "we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum dura…

Amazon builds datacenter redundancy in the same geographic locale. You can then setup synchronous replication between datacenters without atrocious latency, as all the disks are fairly close by, albeit powered [in an emergency] by independent generators. OTOH, Google does datacenter redundancy across different locales, which make synchronous replication perform much worse, like you noted.

GCE appears to have a similar region/zone structure as Amazon, but neither provider replicates block storage across zones.

Re: Google loses data as lightning strikes

#124

Post by mirimir »

nemo1618 wrote: post #118
danepowell wrote: post #94

This raises a relevant concern that's been on my mind: what's the best way to back up cloud services? Given that services like S3 and Google Drive have many more nines of durability than any local storage system I could devise, are backups even worth the trouble? There are a lot of cloud-to-cloud backup services out there, but to me that seems like the blind leading the blind, especially with regards to malicious dat…

If you really care about durability, your best best is erasure-coding + a wide geographic distribution of shards. For example, you could encode 1 TB of data into four shards, each shard containing 500 GB. You distribute these to servers in SF, NYC, Berlin, and Sydney. The key here is that you only need two shards to recover your 1 TB of data, and they can be any two shards. So if lightning strikes Berlin, and the Big…

It's been a while since I followed this. I see over 100 Reed-Solomon erasure-coding projects on GitHub. Which would you recommend?

Re: Google loses data as lightning strikes

#125

Post by nl »

pilsetnieks wrote: post #101
Natsu wrote: post #98

Earlier quoted context omitted.

Ref: http://www.physics.princeton.edu/ph115/LQ.pdf

Every time this story is posted somewhere, I just have to read it end to end.

Relevant: The thermodynamic arrow would reverse during a contracting phase of the Universe or inside black holes.

http://journals.aps.org/prd/abstract/10.1103/PhysRevD.32.248... (S. W. Hawking, 1985 <- Just so you know it isn't a crackpot writing this. It isn't a widely accepted view though)

Re: Google loses data as lightning strikes

#127

Post by pacala »

wmf wrote: post #123
pacala wrote: post #116

Earlier quoted context omitted.

Amazon builds datacenter redundancy in the same geographic locale. You can then setup synchronous replication between datacenters without atrocious latency, as all the disks are fairly close by, albeit powered [in an emergency] by independent generators. OTOH, Google does datacenter redundancy across different locales, which make synchronous replication perform much worse, like you noted.

GCE appears to have a similar region/zone structure as Amazon, but neither provider replicates block storage across zones.

I stand corrected. Looking at https://cloud.google.com/compute/docs/zones?hl=en, it seems that GCE has multiple zones in the same locale (region). I might have confused with Google's internal setup, which during my tenure there, was heavy on using replication across locales, up to systems like Megastore and the 200ms synchronously replicated commit.

Re: Google loses data as lightning strikes

#129

Post by yeukhon »

idlewords wrote: post #24

Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years." I think my favorite part of that is "on average", a…

Well the expectation is different for different players. If Facebook lose 1 photo for 1 customer out of 10M customers, FB wouldn't care. The user may just assume a glitch. For a small business, losing 1 photo for 1 customer out of 10,000 customers can be a big deal, but nonetheless not the worst case. Offer sincere apology, do extra data backup/replication if those data are extremely valuable to customer, and roll out a new service.
Post reply on HN