Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

Since others have already demonstrated why the remainder of your comment is overly simplistic, I'll tackle this bit.

Generally, software releases are not rolled out at a constant rate to all machines. A typical thing to do is to release it to staging, then to a "canary" subset of machines (e.g. to 1% or 5% of the machines).. Once all seems well there (e.g. metrics are clean and the canaries have handled X writes, reads, and simulated drive failures), it can be rolled out to a larger subset, and eventually to all machines.

In that way, the release can take whatever total amount of time is desired while still catching any such bugs fairly reliably.

Ideally, at backlblaze they could ensure that their canary instances are "data-redundancy aware" such that even if the 5% they roll to for the canary test all explode, data is still safe.

Regardless, any talk of "recovering data faster than software releases" is completely silly and totally misses the reality of how releases are done, how recovery is done, and what sort of bugs might happen. The math based on faulty assumptions about rate is also pointless.

  Which means every customer suffers data loss 

The customer only suffers data loss if they lost their 'master' copy of the data as well during the outage. Iff they don't have a secondary backup solution.

> During the 6 days the data would be available it just might have to be reconstructed on the fly by the error correcting rather than read directly.

Correct. More specifically, the FIRST time the data is accessed in any 24 hour period it must ALWAYS be reconstructed from the Reed-Solomon encoded parts on 17 other drives on 17 other machines. Any 17 is fine, so it's totally fine if 1 or 2 drives are not available. Once reconstructed it is stored in a set of front end cache computers that have fast SSDs for this purpose.

The second time the same file is accessed in a 24 hour period, it will be fetched out of the SSD cache layer so it won't even hit the spinning drives and won't care if all 20 drives are offline.

> "primary traffic" (read/write stuff) is prioritized over "rebuild traffic"

Yes. Backblaze balances between the two if only one drive has failed, but as a tome (20 drive group spread across 20 computers) becomes more badly degraded Backblaze begins favoring the rebuild. When two drives have failed out of 20, Backblaze stops allowing any writes to that tome because more writes will tend to fail yet another drive. Fewer writes offloads the tome. But we still allow reads. At Backblaze, we have never been 3 drives degraded out of 20 (knock on wood), but if this ever occurs the 20 drive tome is now running without parity -> so in that case we even stop allowing reads AT ALL until we are returned to at least 1 drive of fully redundant parity.

> spread across 10,000 drives, which is unrealistic

I claim it is also undesirable. Backblaze specifically made the conscious decision that the parts of any one single "large file" (these can be up to 10 TBytes each) are all stored within the same "vault". A vault is 20 computers in 20 separate racks. This allows a single vault to check the consistency and integrity of a large file periodically without communicating to other vaults in the datacenter.

The vaults have been a really good unit of scaling for Backblaze. If the vaults can maintain their performance, then we know we can just stamp out more vaults because there is almost no communication between vaults.