Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

backblaze.com

Post reply on HN
Print view View original

21–30 of 190 posts

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#21

Post by gm-conspiracy »

Johnny555 wrote: post #17
londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

That could be a power surge taking out a whole rack This failure mode, at least, is already accounted for by sharding data across cabinets: Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet. https://www.backblaze.com/blog/vaul…

I thought AZs were in the same physical location, just separate networks, no?

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#22

Post by jsjohnst »

riku_iki wrote: post #10

> if you store 1 million objects in B2 for 10 million years, you would expect to lose 1 file. Can this be reformulated: you store 10 trln objects (e.g. 100TB of 10 byte records), you lose 1 record each year. Also curious what are the stats from other providers.

> Can this be reformulated

Roughly, yes.

> Also curious what are the stats from other providers.

As to other providers, most are 6+ 9s that I’ve looked at, with many in the 8-9 range. Anything over 8 is (as they admitted) essentially marketing porn and not a useful metric (for reasons they mentioned as well as ones said by other comments here).

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#23

Post by coderholic »

gm-conspiracy wrote: post #21
Johnny555 wrote: post #17

Earlier quoted context omitted.

That could be a power surge taking out a whole rack This failure mode, at least, is already accounted for by sharding data across cabinets: Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet. https://www.backblaze.com/blog/vaul…

I thought AZs were in the same physical location, just separate networks, no?

No, they're different physical locations.

From https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Conce...:

> Each AWS Region has multiple, isolated locations known as Availability Zones.

And from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-re...:

> Each region is completely independent. Each Availability Zone is isolated, but the Availability Zones in a region are connected through low-latency links.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#24

Post by jsjohnst »

gm-conspiracy wrote: post #21
Johnny555 wrote: post #17

Earlier quoted context omitted.

That could be a power surge taking out a whole rack This failure mode, at least, is already accounted for by sharding data across cabinets: Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet. https://www.backblaze.com/blog/vaul…

I thought AZs were in the same physical location, just separate networks, no?

Nope. They aren’t even in the same city. For example, each of the AZs in us-west-2 are separated by about 50-60 miles.

This doesn’t list all the locations, but is a good map to get an idea:

https://www.google.com/maps/d/u/0/viewer?ll=50.9584270000000...

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#25

Post by riobard »

jsjohnst wrote: post #20
londons_explore wrote: post #12

I've very disappointed their recovery time is 6 days! Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed evenly. In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. Maybe 10 seconds with some sluggish tail machines. Why do you need it done i…

> In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. I want to know where you can find a drive that can write 12TB/sec of data! (In other words, you clearly missed half the problem. To add a new replacement drive, you have to be able to write to it the data from an original drive. Also RS code…

Additionally, it implies that data needs to be spread across 10,000 drives, which is unrealistic anyway.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#26

Post by ChrisLomont »

londons_explore wrote: post #9

This analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when fail…

>The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-(

You just made the same mistake you're criticizing. You assumed the 100K files were uniformly and independently spread. They're also likely clustered, and perhaps not even at the same data center. Given the variety of drives Backblaze uses, the drives are also not likely to all be the same model, so your failure method is also unlikely.

You also only looked at the case there are exactly 16 good drives. The proper failure estimate is 1 - (odds of 17 good + odds of 18 + odds of 19 + odds of 20). I'm not sure where you got the 20 x 19 x 18 part either. Did you mean 30 choose 16 or something like that? Using the proper 1-... method I get 0.00267, not 0.03.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#27

Post by chx »

Animats wrote: post #18

Financial failure or service shutdown by the provider is the highest risk for long term storage. The backup services CrashPlan, Dell DataSafe, Symantec, Ubuntu One, and Nirvanix all shut down. Nirvanix only gave two weeks notice for users to save their data.[1] [1] https://www.computerweekly.com/opinion/Nirvanix-failure-a-bl...

Just for fun, here's my report of Nirvanix in 2008. I feel no problems sharing it since both NowPublic and Nirvanix is long gone. Let's just say there were reasons they went under:

1) Sideloading. I was unable to benchmark or even to get this to work. Images requested to be loaded from media-src.nowpublic.com to node4 @ Nirvanix never showed up. I showed my code to [nirvanixcontact] who said that the code looked OK but someone did a load testing on node4 without informing Nirvanix and that steps are made so that such a situation won't occur again. He told me that the requests are not lsot but they still have not landed. Again, my code can be at fault and I would be happy to run a sideload example code.

2) Upload speeds. I have uploaded from d2 to Nirvanix 100 images each between 180-8584 kbytes totalling almost exactly 100 MB (101 844 931 bytes). The upload was a single HTTP request. The uploads took 18-19 minutes (I repeated the experiment). To give us a comparison I changed the URL in the script to a one line PHP script on another server (at hostignition.com) which just echo'd the number of uploaded files. This took 16.25 seconds and echo'd 100 so seemingly the files landed.

2a) I tried to get another node via the LoginProxy method which we would need for uploads anyways. While LoginProxy itself did work, GetStorageNodeExtended https://services.nirvanix.com/ws/IMFS/GetStorageNodeExtended... always fails with ResponseCode 80006, ErrorMessage: Session not found for ip = 67.15.102.70, token = e7b00d25-fc35-431c-9437-9a4302767f46. Seemingly, does not pick up the consumerIP.

3) The image conversion itself is blazing fast though. These images took a total of 101.58 seconds to convert and this includes 300 HTTP requests (200 sent to d2 from Nirvanix, 100 to Nirvanix).

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#28

Post by Johnny555 »

gm-conspiracy wrote: post #21
Johnny555 wrote: post #17

Earlier quoted context omitted.

That could be a power surge taking out a whole rack This failure mode, at least, is already accounted for by sharding data across cabinets: Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet. https://www.backblaze.com/blog/vaul…

I thought AZs were in the same physical location, just separate networks, no?

They are separate datacenters, I don't think they make any promises about how far they are apart from each other, but at least in some regions they are 10+ miles apart.

The AWS Cloud infrastructure is built around AWS Regions and Availability Zones. An AWS Region is a physical location in the world where we have multiple Availability Zones. Availability Zones consist of one or more discrete data centers, each with redundant power, networking, and connectivity, housed in separate facilities

https://docs.aws.amazon.com/aws-technical-content/latest/aws...

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#29

Post by riku_iki »

jsjohnst wrote: post #22
riku_iki wrote: post #10

> if you store 1 million objects in B2 for 10 million years, you would expect to lose 1 file. Can this be reformulated: you store 10 trln objects (e.g. 100TB of 10 byte records), you lose 1 record each year. Also curious what are the stats from other providers.

> Can this be reformulated Roughly, yes. > Also curious what are the stats from other providers. As to other providers, most are 6+ 9s that I’ve looked at, with many in the 8-9 range. Anything over 8 is (as they admitted) essentially marketing porn and not a useful metric (for reasons they mentioned as well as ones said by other comments here).

And yet, no much complains about lost GMail letters in internet because of disc failures.. And GMail is likely stores much more than 10 trln objects and 100TB of data.
Post reply on HN