Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google loses data as lightning strikes

bbc.com

Post reply on HN
Print view View original

131–140 of 142 posts

Re: Google loses data as lightning strikes

#131

Post by okadaka »

So Google said: "...although... the storage systems are designed with battery backup, some recently written data was located on storage systems which were more susceptible to power failure from extended or repeated battery drain. In almost all cases the data was successfully committed to stable storage, although manual intervention was required in order to restore the systems to their normal serving state. However, in a very few cases, recent writes were unrecoverable, leading to permanent data loss on the Persistent Disk."

I thought battery is supposed to cover writing the entire write buffer cache to disk in case of power loss. Sounds like they had some badly designed gear which did no account for partial battery charge which should downsize the cache to battery's capacity.

Re: Google loses data as lightning strikes

#132

Post by sbecrab4 »

tedchs wrote: post #75

Key point from Google's status incident page that clarifies some of the partially-incorrect statements in the press and comments about this: "In a very small fraction of cases (less than 0.000001% of PD space in europe-west1-b), there was permanent data loss." https://status.cloud.google.com/incident/compute/15056#57195...

Quoting the data lost as a percentage of disk space is both accurate and misleading. It makes the impact sound tiny because only recent writes were affected. Obviously writes that were in flight at the time of the incident are going to be a tiny percentage of overall storage. What they don't tell us is what percentage of persistent disks which were in use at the time were affected. That percentage is likely far higher. If only 0.000001% of volumes in use were affected it would never have made the news.

Re: Google loses data as lightning strikes

#136

Post by EduardoBautista »

cjensen wrote: post #112
onethumb wrote: post #105

Earlier quoted context omitted.

How so? We'd need to know both how much data cjensen lost and how much data Glacier has in it to tell, neither of which I know from this conversation.

My recollection: around 20GB... however , they clearly were willing to try to get some of it back if that was helpful. It was just a redundant offsite backup for me, so there was no need.

Would have been interesting to see if they had been able to.

Re: Google loses data as lightning strikes

#137

Post by fho »

pilsetnieks wrote: post #102
tsunamifury wrote: post #89

Earlier quoted context omitted.

Funny enough, its funded by Bezos

Bezos isn't the only one. The Long Now foundation ( http://longnow.org ) has some other interesting projects besides the clock. If you like this kind of thing, I'd suggest reading Neal Stephenson's Anathem . It might be hard to get into it, it might be his most impenetrable book yet but it really pays off later.

Several of Neal Stephenson's books are like this ... I finished the Cryptonomicon earlier this year and have just started to read REAMDE. I often find myself reading the same sentence several times because they are so convoluted ...

Re: Google loses data as lightning strikes

#139

Post by nemo1618 »

mirimir wrote: post #124
nemo1618 wrote: post #118

Earlier quoted context omitted.

If you really care about durability, your best best is erasure-coding + a wide geographic distribution of shards. For example, you could encode 1 TB of data into four shards, each shard containing 500 GB. You distribute these to servers in SF, NYC, Berlin, and Sydney. The key here is that you only need two shards to recover your 1 TB of data, and they can be any two shards. So if lightning strikes Berlin, and the Big…

It's been a while since I followed this. I see over 100 Reed-Solomon erasure-coding projects on GitHub. Which would you recommend?

Well, I have to shamelessly plug my startup, www.siacoin.com, which implements the scheme I described above using a peer-to-peer network, with payments made on a blockchain. We are using Klaus Post's excellent pure-Go implementation (https://github.com/klauspost/reedsolomon) which exceeds 1GB/s throughput.

Re: Google loses data as lightning strikes

#140

Post by mirimir »

nemo1618 wrote: post #139
mirimir wrote: post #124

Earlier quoted context omitted.

It's been a while since I followed this. I see over 100 Reed-Solomon erasure-coding projects on GitHub. Which would you recommend?

Well, I have to shamelessly plug my startup, www.siacoin.com, which implements the scheme I described above using a peer-to-peer network, with payments made on a blockchain. We are using Klaus Post's excellent pure-Go implementation ( https://github.com/klauspost/reedsolomon ) which exceeds 1GB/s throughput.

Thank you. It's very cool. But can Sia work with hosts that are reachable only as Tor hidden services?
Post reply on HN