21–30 of 142 posts
> Google said that just 0.000001% of disk space was permanently affected. Assuming 1 petabyte of total storage at the datacenter, that equates to about 100mb. I wonder how much storage they have there.
Earlier quoted context omitted.
The incident report ismavis posted below (and linked in the article) has far more information: https://status.cloud.google.com/incident/compute/15056#57195...
[deleted]
I think the reference to batteries would more likely be in reference to a DRUPS (Diesel Rotary UPS) which most datacenters run, sometimes with some form of battery in combination with the flywheel. Typically the combination of kinetic energy in the flywheel (and potentially batteries) only hold enough power to last for 30 seconds or so (often as low as 10 seconds), which gives the diesel power generators enough time to come online and take over from there.
What I guess is that might have happened is that grid power was lost, they switched over to UPS fine the first time, grid power came back so they swapped back, repeat a few times and on one of those times the batteries didn't have enough charge to keep things going for the generator swap-over.
This could potentially explain a lingering error on my Google Drive. Might there be movement within Google to contact the owners of data that was lost?
(Disclaimer: I work there but have absolutely no knowledge about how Drive stores data)
I think my favorite part of that is "on average", as if you will be making repeated ten-million-year trials of this effectively brand new technology.
The point is that once you get into several nines of reliability, really rare events that are impossible to model start to dominate your risk budget.
Beyond security, this highlights one of the main issues with the cloud. Was there no backup? Of course once you get beyond the headline, I think most people are much worse with protecting themselves from rare outages than Google.
As mentioned in the article, it only affected recently written data of 'Google Compute Engine' services. GCE allows user to launch VMs and generate arbitrary data on the server. Normally, Google redundantly distributes out data to at least 3 different geographically distinct locations. Check out the 'BigTable' white paper [0] for more info. For 99% of cases (and pretty well all user cases), this would not cause data…
Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years." I think my favorite part of that is "on average", a…
So Google just exhausted their 11 9's for centuries to come.
Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years." I think my favorite part of that is "on average", a…
To accurately compare them you'd need to look at AWS EBS: "Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complete or partial loss of the volume, depending on the size and performance of the volume."