151–160 of 190 posts
Earlier quoted context omitted.
If Google suspends your account for background music playing in a YouTube video, you might still lose access to your files in Google drive / cloud - even if the files are encrypted.
You should compartmentalize those.
(this isn't quite true in my case, but Google did go to some lebghts to merge yt accounts into Google accounts recently).
Earlier quoted context omitted.
Which means every customer suffers data loss They suffer partial backup loss. The customer only suffers data loss if they lost their 'master' copy of the data as well during the outage. Iff they don't have a secondary backup solution.
Are they positioning their "B2" product solely as a backup solution, rather than something closer to an S3 competitor? If it's the latter, "data loss" seems an appropriate characterization.
This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…
I'm not sure how anyone, in their disaster recovery plans, ever expect anything less than 100% failure of a data center. The scenario is: a tornado hits the data center. Your power outage causing 3% of drives to fail is just a subset of that.
This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…
« The chances that you only have 16 good drives for a file is about (0.97^16 ∗ 0.03^4)∗20∗19∗18 = 0.3% Yet your customer has 100k files! The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-( » Your math is completely wrong. In reality 92% of customers suffer no data loss. The chance of a file having 4 failed drives (16 good drives) is: .03^4 = 0.000081…
Earlier quoted context omitted.
Isn't the expected life span of the company more limiting? Plenty of cloud storage companies go out of business (typically, they run out of money). You can apply Gott's law to this. It's pretty grim
I guess this is what you're referring to: https://en.wikipedia.org/wiki/J._Richard_Gott#Copernicus_met...
[t/3, 3t] with 50% confidence
[t/4, 4t] with 60% confidence
[t/39, 39t] with 95% confidenceEarlier quoted context omitted.
I wonder if there's a general term in engineering for the case where a particular risk has been reduced well below the likelihood of more serious but exotic risks. I've heard about this most in cryptography where we can sometimes say that the risk of, say, a system failing due to an unexpected hash collision is drastically less than the risk of the system failing due to cosmic radiation or various natural disasters.…
It’s not a perfect match, but Rumsfeldian “unknown unknowns” come to mind. Specifically: every X-nines durability design will be compromised by some failure mode you didn’t think of.
Earlier quoted context omitted.
That could be a power surge taking out a whole rack This failure mode, at least, is already accounted for by sharding data across cabinets: Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet. https://www.backblaze.com/blog/vaul…
A better example would be the ceramic bearing fiasco that NetApp experienced with Seagate. Seagate had switched to a floating ceramic bearing on one family of their fiber channel drives. In those drives one or more of the bearings would shatter and start spreading ceramic dust across the disk surface. This happened between 3 and 4 years of run time and the disk would rapidly fail after that happened. People that boug…
Earlier quoted context omitted.
« The chances that you only have 16 good drives for a file is about (0.97^16 ∗ 0.03^4)∗20∗19∗18 = 0.3% Yet your customer has 100k files! The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-( » Your math is completely wrong. In reality 92% of customers suffer no data loss. The chance of a file having 4 failed drives (16 good drives) is: .03^4 = 0.000081…
The original maths is correct. You have neglected the fact that the 4 failed drives can be any of the 20. Your calculations are for 4 specific drives failing.
The OP still has a slight error though. C(20,4) = 4845 possible combinations of failing drives (4 out of 20.)
Therefore the chance a file is irrecoverable is: .97^16 × .03^4 × 4845 = 0.241% (not 0.3%)
But the OP's conclusion is largely correct: every customer will have some irrecoverable files.
Earlier quoted context omitted.
I've been doing a poor man's version of this for home videos using 3 hard drives and rsync. It's easy to replace a drive and they are not likely to go out at the same time. But one thing that bugs me is that unless the drive fails hard (e.g. noticed by SMART or unable to read at all) how do I know the data on the drive is not corrupted without reading it? Are there best practices to continuously compare the replicas…
> how do I know the data on the drive is not corrupted without reading it? Are there best practices to continuously compare the replicas in the background? I assume you're talking about already-written sectors becoming unreadable or a similar failure. Unfortunately, I don't think you can. This is what I believe the "patrol read" feature of RAID cards is meant to address. Fortunately, however, I don't believe there's…
Comparison is needed to address misdirected writes and bit rot in the very least, see "An Analysis of Data Corruption in the Storage Stack" [1]. You can't count on your drive firmware or RAID firmware to get this right. You need bigger end-to-end checksums, and you need to scrub.
Earlier quoted context omitted.
> Correlated failures are common in drives. This is why, when I was building DIY arrays for startups (around the same time Backblaze published their first pod design [1]), I went through the extra effort of sourcing disks from as many different vendors as possible. Although it was somewhat more time consuming and limited how good a price I could get and how fast the delivery could be, it meant that, for any given dis…
I've been doing a poor man's version of this for home videos using 3 hard drives and rsync. It's easy to replace a drive and they are not likely to go out at the same time. But one thing that bugs me is that unless the drive fails hard (e.g. noticed by SMART or unable to read at all) how do I know the data on the drive is not corrupted without reading it? Are there best practices to continuously compare the replicas…