Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

backblaze.com

Post reply on HN
Print view View original

181–190 of 190 posts

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#181

Post by mmt »

walterbell wrote: post #160
foobarian wrote: post #140

Earlier quoted context omitted.

I've been doing a poor man's version of this for home videos using 3 hard drives and rsync. It's easy to replace a drive and they are not likely to go out at the same time. But one thing that bugs me is that unless the drive fails hard (e.g. noticed by SMART or unable to read at all) how do I know the data on the drive is not corrupted without reading it? Are there best practices to continuously compare the replicas…

Use ZFS, it can perform periodic integrity checks.

I've never done so outside of FreeNAS appliances, partly because I remain persuaded that offloading the RAID portion to a card is more cost-effective and higher-performance, especially on otherwise RAM- and/or IO-constrained servers, and partly because the ZFS support under Linux has, historically, been less than ideal.

Higher-level checksum failures are, however, a situation, where I would appreciate an integration between filesystem and RAID, as I'd want a checksum error to mark a drive as bad, just like any other read error.

Do you happen to know if ZFS does that?

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#182

Post by atYevP »

u02sgb wrote: post #162
atYevP wrote: post #105

Earlier quoted context omitted.

Yev from Backblaze here -> I believe we do send SMSs in the case of a Cap or Alert getting reached, so yes that could be possible - though I'm not sure if an SMS is part of our billing failure process - that's an interesting question!

Backblaze B2 customer here. My credit card stopped accepting your billing and no SMS for me. Took me a month or so to notice the emails and update my details. I've got SMS alerts active for Caps. Would be worth adding that as was a bit scary when I noticed the mail (think it was the third one you'd sent!).

VERY good to know! Thank you!

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#183

Post by earenndil »

kalleboo wrote: post #147
the8472 wrote: post #122

Earlier quoted context omitted.

You should compartmentalize those.

That can be difficult to maintain though, as Google is pretty eager to link accounts. You have horror stories like https://www.reddit.com/r/tifu/comments/8kvias/tifu_by_gettin... > Eventually someone realized that their non-work accounts were banned as well. It wasn't until yesterday that someone made the connection. Anyone who had their accounts as a recovery option were also caught in the ban wave.

I'm pretty sure that was found to fake, no?

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#184

Post by brianwski »

k__ wrote: post #95
brianwski wrote: post #58

Earlier quoted context omitted.

Disclaimer: I'm the author of the blog post. :-) > well written and refreshingly transparent Thank you! In the interests of full transparency, the blog post was a collaborative affair and was proof read and edited for clarity by several people at Backblaze. > discussion about why it doesn't matter One of the philosophies Backblaze uses is to build a reliable component out of several inexpensive and unrelated componen…

BB and S3 both have eleven 9 durability, how much does using both increase this?

> BB and S3 both have eleven 9 durability, how much does using both increase this?

Putting your data in either Backblaze B2 or Amazon S3 suffer from other failure modes outside of the durability of the raw system. For example, let's say your IT person is poking around in their Amazon S3 account and accidentally clicks the "delete" button and all your data is gone? Or what if your credit card has a transaction declined, and your IT guy has left your company or the emails from Amazon are being put in the "Spam" folder of your email program. Or maybe a malicious Amazon employee writes a program to delete all the data in Amazon S3 from all customers? What if one of your employees is really disgruntled and logs into your Amazon S3 account and just to spite you deletes all your data?

In every one of these situations, if you have a copy in Backblaze B2 and also another copy in Amazon S3, you can recover your data from the other vendor.

I recommend using a separate credit card to pay for your Amazon S3 account and your Backblaze B2 account. They should expire a year apart. And don't give the logins to both systems to one disgruntled employee in your organization. Only give that disgruntled employee access to one or the other.

Make sense?

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#185

Post by e12e »

stefan_ wrote: post #180
e12e wrote: post #119

Earlier quoted context omitted.

While I'm sceptical of content filters, even with a home search, it seems unlikely you'd end up in cuffs unless a) the filter caught acymtual illegal content, or b) the search turned up something illegal. You might get killed in the course of the initial police raid though..

Well you take a picture of your kid in the bathtub, now who can tell the difference?

If pictures of naked kids is illegal in your jurisdiction, you've got bigger problems. I guess that's true for some locations, though. Still, nude people =! pornography.

(kids below age of consent sexting each other is another, related, problem)

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#186

Post by pas »

voidmain wrote: post #124
brianwski wrote: post #36

Earlier quoted context omitted.

Disclaimer: I'm the author of the blog post. :-) > some of the assumptions that Brian was tossing out are not good ones to make. We COMPLETELY welcome other analysis and listing other assumptions. Internally, we argued endlessly about why this or that wasn't totally accurate, and finally decided to publish the math WITH all of our assumptions exposed so you could be the judge. If Amazon wants to publish their assumpt…

> If Amazon wants to publish their assumptions for S3 for comparison, we're all ears. The 2010 S3 calculation is obviously also wrong! I totally feel your pain in terms of wanting to have a directly comparable answer, but the reasons you give that "it doesn't matter" (and others, including correlated faults, software bugs, model risk, and security breaches) are actually reasons why the stated durability number is wro…

As far as I know, if they claim multi AZ in a city (region), that really means at least separate buildings, but likely separate locations withing the region too.

Though I'd welcome hard data on this very much.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#187

Post by mrb »

mrb wrote: post #173
majewsky wrote: post #170

Earlier quoted context omitted.

I think you're not considering that failed drives will be replaced and the data on them reconstructed from the other shards. This failure mode requires 4 of 20 drives to fail in such a short amount of time that reconstruction cannot be completed.

Yes, but this was the OP's scenario: what happens when 3% of drives all fail at the same time when the DC is powered back on. Edit: actually the math is still wrong. The chance any 4 out of 20 drive is failing is: .03^4 × C(20,4) = .03^4 × 4845 = 0.392% — There is no need to multiply by .97^16 as the status of the other 16 drives is irrelevant.

Decidedly, statistics is hard. Everything above is wrong. Let's label the twenty drives D0 through D19. There are 2^20 possible scenarios, which can be represented as a string of 20 bits:

• 00000000000000000000 = all 20 drives are working

• 00000000000000000001 = D19-D1 working, D0 failing

• 00000000000000000010 = D19-D2 working, D1 failing, D0 working

• 00000000000000000011 = D19-D2 working, D1-D0 failing

• etc

The probability of each of these scenarios is:

• 00000000000000000000: .97^20

• 00000000000000000001: .97^19 × .03

• 00000000000000000010: .97^19 × .03

• 00000000000000000011: .97^18 × .03^2

• etc

There are C(20,4) = 4845 scenarios with exactly four failing drives (four "1" bits.) The probability of each scenario is .97^16 × .03^4. Therefore the probability of 4 failing drives (any drive) is the sum of the probability of each scenario: .97^16 × .03^4 × C(20,4) like I said 3 comments above.

However the probability of a file being irrecoverable is P(4 failing drives) + P(5 failing drives) + ... + P(20 failing drives):

    .97^16 × .03^4 × C(20,4)
  + .97^15 × .03^5 × C(20,5)
  + ...
  + .97^0 × .03^20 × C(20,20)
  = 0.267%

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#188

Post by walterbell »

mmt wrote: post #181
walterbell wrote: post #160

Earlier quoted context omitted.

Use ZFS, it can perform periodic integrity checks.

I've never done so outside of FreeNAS appliances, partly because I remain persuaded that offloading the RAID portion to a card is more cost-effective and higher-performance, especially on otherwise RAM- and/or IO-constrained servers, and partly because the ZFS support under Linux has, historically, been less than ideal. Higher-level checksum failures are, however, a situation, where I would appreciate an integration…

Does this help?

https://prefetch.net/blog/2011/10/15/using-the-zfs-scrub-fea...

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#189

Post by mmt »

walterbell wrote: post #188
mmt wrote: post #181

Earlier quoted context omitted.

I've never done so outside of FreeNAS appliances, partly because I remain persuaded that offloading the RAID portion to a card is more cost-effective and higher-performance, especially on otherwise RAM- and/or IO-constrained servers, and partly because the ZFS support under Linux has, historically, been less than ideal. Higher-level checksum failures are, however, a situation, where I would appreciate an integration…

Does this help? https://prefetch.net/blog/2011/10/15/using-the-zfs-scrub-fea...

Unfortunately, no, it doesn't really say how ZFS behaves when an error is encountered.

This is super-disturbing and a dealbreaker, if it's still true:

> The scrub operation will consume any and all I/O resources on the system (there are supposed to be throttles in place, but I’ve yet to see them work effectively), so you definitely want to run it when you’re system isn’t busy servicing your customers.

I browsed a little of the oracle.com ZFS documentation but couldn't find much in the way of what triggers it to decide that a device is "faulted" other than being totally unreachable.

Re: Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter

#190

Post by juancn »

My experience is that it's bullshit. I had a backup (damn I still have one) with Backblaze, when attempting to restore it, maybe 30% of the files survived restore. The rest are lost in smoke.

They don't have any way to detect corruption in the data or if they have, the backup clients are oblivious to it.

I lost about a 150GB of family photos and videos.

Post reply on HN