At Amazon, when resolving an issue in our internal ticketing system I recall there being "Act of God" as a reason code. Seems applicable here.
Google loses data as lightning strikes
91–100 of 142 posts
Re: Google loses data as lightning strikes
#92Earlier quoted context omitted.
[deleted]
AFAIK google doesn't use hardware-based arrays in their servers. I think the reference to batteries would more likely be in reference to a DRUPS (Diesel Rotary UPS) which most datacenters run, sometimes with some form of battery in combination with the flywheel. Typically the combination of kinetic energy in the flywheel (and potentially batteries) only hold enough power to last for 30 seconds or so (often as low as…
A so UPS on the cheap a real UPS is always running of the battery Telco style and you are testing your gensets at least weakly and checking the tanks every month or so.
At BT they where tested every day for the really sensitive systems.
Re: Google loses data as lightning strikes
#93I work on cell sites, grounding system design and repair is a primary design element, even then, the presumption in the industry is that if a site takes a direct hit - or for that matter a nearby strike - the equipment is a total loss. The surge suppression gear we put in (lead ins at power feeds, RF feed, etc) is mostly to prevent a fire and to ensure the extra energy goes largely to ground.. but it won't prevent de…
Are you saying essentially that "There is no such thing as a surge protector, they don't physically exist. Only surge reducers exist." Because that's what it sounds like to me.
EDIT:
All right, I'll rephrase. According to Google's infobox from nat'l geographic, lightning generates up to 1 billion volts.
-> Are surge protectors at even the highest-end data centers simply not rated to a billion volts of surge protection?
Re: Google loses data as lightning strikes
#94This raises a relevant concern that's been on my mind: what's the best way to back up cloud services? Given that services like S3 and Google Drive have many more nines of durability than any local storage system I could devise, are backups even worth the trouble?
There are a lot of cloud-to-cloud backup services out there, but to me that seems like the blind leading the blind, especially with regards to malicious data destruction. For instance, I've recently been experimenting with Cloudally to automatically back up Google Drive, which seems like a good solution at first- until you think about the fact that Cloudally uses Google accounts for authentication (and doesn't use 2FA for native authentication). In other words, an attacker with access to my primary data (Google Drive) would also have access to my backups. Worse than that, Cloudally actually increases the attack surface, since its lack of 2FA presumably makes it easier to crack than my Google account.
Similarly, I'm guessing a lot of cloud backup services share data centers with the services they are backing up.
Re: Google loses data as lightning strikes
#95It sounds like they mostly lost recently created/stored data that hadn't yet been fully replicated to the required degree of redundancy.
Re: Google loses data as lightning strikes
#96Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years." I think my favorite part of that is "on average", a…
I always think of Feynman's minority report [0] on the Challenger Disaster when these types of risk assessments pop up.
[0] http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roger...
Re: Google loses data as lightning strikes
#97This raises a relevant concern that's been on my mind: what's the best way to back up cloud services? Given that services like S3 and Google Drive have many more nines of durability than any local storage system I could devise, are backups even worth the trouble? There are a lot of cloud-to-cloud backup services out there, but to me that seems like the blind leading the blind, especially with regards to malicious dat…
durability is not same as "protection". It doesn't include items like government seizure, hacking. Your own drives, in your physical possession fill in that gap (somewhat).
Re: Google loses data as lightning strikes
#98Earlier quoted context omitted.
Keeping one object for a zillion years sounds like an interesting project, now that you bring it up. Our data integrity is such that it will survive universal cataclysm and the reduction of all data to a single point of infinite energy.[1] [1]guarantee applies to a single bit of data[2] [2]bit value must be 1
THERE IS AS YET INSUFFICIENT DATA FOR A MEANINGFUL ANSWER
Re: Google loses data as lightning strikes
#99Earlier quoted context omitted.
Nowadays they use Reed–Solomon coding to effectively distribute their data without copying it to 3 places.
Do you have a source for that? Why would they start using it now?
I can't find the presentation anywhere, but at HBaseCon 2014 one of the lead developers of Bigtable stated that they went to RS. Even for their older databases.
EDIT: In this video https://vimeo.com/100153741, around the 23 minute mark
Re: Google loses data as lightning strikes
#100Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years." I think my favorite part of that is "on average", a…
That's one of the reasons that I'm excited about decentralized storage. Data stored on dozens or hundreds of nodes spread across multiple countries and jurisdictions is much more robust to things like earthquakes, storms, government intervention, and companies going out of business or changing their profitability model. It has a much stronger defence against black swan events. When you are storing many billions of fi…
I'm quite hyped about ipfs. I think they can deliver in few years.