31–40 of 142 posts
Beyond security, this highlights one of the main issues with the cloud. Was there no backup? Of course once you get beyond the headline, I think most people are much worse with protecting themselves from rare outages than Google.
Earlier quoted context omitted.
As mentioned in the article, it only affected recently written data of 'Google Compute Engine' services. GCE allows user to launch VMs and generate arbitrary data on the server. Normally, Google redundantly distributes out data to at least 3 different geographically distinct locations. Check out the 'BigTable' white paper [0] for more info. For 99% of cases (and pretty well all user cases), this would not cause data…
Nowadays they use Reed–Solomon coding to effectively distribute their data without copying it to 3 places.
Beyond security, this highlights one of the main issues with the cloud. Was there no backup? Of course once you get beyond the headline, I think most people are much worse with protecting themselves from rare outages than Google.
Google Compute Engine offers customers the option to make snapshots for backup, or use a true "cloud" storage engine. If anyone lost data here it was customers explicitly not doing backups and only using a single zone. I don't know why anyone would expect different. GCE easily allows you to network machines in multiple data centers, but close geographically. So you'd only need to handle region-wide disasters.
Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years." I think my favorite part of that is "on average", a…
"just 0.000001% of disk space was permanently affected." So Google just exhausted their 11 9's for centuries to come.
Curious - how do they know lightning hit four times? Was someone outside counting?
This is how, for example, you can know whether a wildfire was started by lightning - once a point of origin is determined, simply check the data for strikes.
https://en.wikipedia.org/wiki/Lightning_detection http://www.lightningmaps.org/
Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years." I think my favorite part of that is "on average", a…
Your main point (that making high reliability systems more conventionally reliable is building your high wall yet higher still) is definitely valid. But the lossage rate is actually a meaningful number given the extremely large number of objects stored in S3.
Curious - how do they know lightning hit four times? Was someone outside counting?
Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years." I think my favorite part of that is "on average", a…
This is S3 which isn't comparable to Google's persistent disks. S3 is equivalent to Google Cloud Storage which has "99.999999999%" durability as per https://cloud.google.com/storage/ To accurately compare them you'd need to look at AWS EBS: "Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complete or partial loss of the volume, depending on the size a…
Earlier quoted context omitted.
[deleted]
AFAIK google doesn't use hardware-based arrays in their servers. I think the reference to batteries would more likely be in reference to a DRUPS (Diesel Rotary UPS) which most datacenters run, sometimes with some form of battery in combination with the flywheel. Typically the combination of kinetic energy in the flywheel (and potentially batteries) only hold enough power to last for 30 seconds or so (often as low as…
