Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Examining Btrfs

arstechnica.com

Post reply on HN
Print view View original

101–110 of 112 posts

Re: Examining Btrfs

#101

Post by bluGill »

wtallis wrote: post #100
bluGill wrote: post #95

Earlier quoted context omitted.

The smallest number you can safely go is 3 at a time - put them in a mirror configuration. This is expensive for the amount of storage you get, so few people are willing to spend that much. Once you want to be cost effective 6-8 is the sweet spot - enough drives for a good amount of usable space, but not so many that multiple drive failures are likely to compromise the array. Nobody is stopping you from other configu…

> The smallest number you can safely go is 3 at a time - put them in a mirror configuration. This is expensive for the amount of storage you get, so few people are willing to spend that much. You're talking in circles. You're assuming the limitations of ZFS, then providing advice based on those limitations, then declaring that there's no advisable use case for a filesystem that's free of those limitations. On btrfs,…

It is not safe to have more than 8 drives in an array. I'm not willing to trust I will never want to stay that small.

Re: Examining Btrfs

#102

Post by wtallis »

bluGill wrote: post #101
wtallis wrote: post #100

Earlier quoted context omitted.

> The smallest number you can safely go is 3 at a time - put them in a mirror configuration. This is expensive for the amount of storage you get, so few people are willing to spend that much. You're talking in circles. You're assuming the limitations of ZFS, then providing advice based on those limitations, then declaring that there's no advisable use case for a filesystem that's free of those limitations. On btrfs,…

It is not safe to have more than 8 drives in an array. I'm not willing to trust I will never want to stay that small.

> It is not safe to have more than 8 drives in an array.

What are you basing this claim on? Did you mean to qualify that statement to only apply to certain RAID modes, or are you again just repeating advice specific to ZFS as if it applies to all storage systems?

Re: Examining Btrfs

#103

Post by bluGill »

wtallis wrote: post #102
bluGill wrote: post #101

Earlier quoted context omitted.

It is not safe to have more than 8 drives in an array. I'm not willing to trust I will never want to stay that small.

> It is not safe to have more than 8 drives in an array. What are you basing this claim on? Did you mean to qualify that statement to only apply to certain RAID modes, or are you again just repeating advice specific to ZFS as if it applies to all storage systems?

All raid. Your odds of a dual fail failure are too high.

Note, good raid6 should have x and y stripes with no more than 8 drives in any stripe. I have no idea what they do, so maybe they can restripe somehow on one drive. Hiding the implementation detail is good if they do this right. But I don't know how to tell if any do it right.

Re: Examining Btrfs

#104

Post by bluGill »

Dylan16807 wrote: post #97
bluGill wrote: post #95

Earlier quoted context omitted.

The smallest number you can safely go is 3 at a time - put them in a mirror configuration. This is expensive for the amount of storage you get, so few people are willing to spend that much. Once you want to be cost effective 6-8 is the sweet spot - enough drives for a good amount of usable space, but not so many that multiple drive failures are likely to compromise the array. Nobody is stopping you from other configu…

I want to work up to 8 drives in a vdev as my storage needs grow, because that's cost-effective. But I don't want to spend 8 drives of money up front, because it's a lot and because drive prices drop over time. Am I not a real use case?

For the first 8 drives, but after that no. BRTFS-raid2 seems to be good for up to 8 drives if that is your use case. But there is no easy upgrade path to that 9th drive. You should not be running less than 3 drives for any data you care about.

Fortunately, most people don't have data they care about. Their photos and documents are on cloud providers like google/dropbox/facebook (and dozens of others I don't even know of). If disaster happens they might lose a couple days work. There is a lot to not like about cloud providers, but they are probably better than what the average person does - no backups at all. Note that the RAID system I've described here is for data integrity, but there are still house burning down cases where you lose it all,

Re: Examining Btrfs

#105

Post by wtallis »

bluGill wrote: post #103
wtallis wrote: post #102

Earlier quoted context omitted.

> It is not safe to have more than 8 drives in an array. What are you basing this claim on? Did you mean to qualify that statement to only apply to certain RAID modes, or are you again just repeating advice specific to ZFS as if it applies to all storage systems?

All raid. Your odds of a dual fail failure are too high. Note, good raid6 should have x and y stripes with no more than 8 drives in any stripe. I have no idea what they do, so maybe they can restripe somehow on one drive. Hiding the implementation detail is good if they do this right. But I don't know how to tell if any do it right.

> All raid. Your odds of a dual fail failure are too high.

These two sentences are in conflict. There are RAID modes where the failure of two drives is recoverable. You probably know this, since you mention one of them in your next sentence. But maybe you don't realize that RAID 6 isn't the only way for an array to survive the failure of two drives, especially if the failures aren't simultaneous. You should read up on some storage systems beyond traditional hardware RAID and ZFS; there are quite a few interesting ideas out there you should open your mind to. Btrfs would be a great place to start your investigation.

Re: Examining Btrfs

#106

Post by Dylan16807 »

bluGill wrote: post #103
wtallis wrote: post #102

Earlier quoted context omitted.

> It is not safe to have more than 8 drives in an array. What are you basing this claim on? Did you mean to qualify that statement to only apply to certain RAID modes, or are you again just repeating advice specific to ZFS as if it applies to all storage systems?

All raid. Your odds of a dual fail failure are too high. Note, good raid6 should have x and y stripes with no more than 8 drives in any stripe. I have no idea what they do, so maybe they can restripe somehow on one drive. Hiding the implementation detail is good if they do this right. But I don't know how to tell if any do it right.

If 8 drives is safe with dual parity, then significantly more than 8 drives would be safe with triple parity, right?

Re: Examining Btrfs

#107

Post by wtallis »

bluGill wrote: post #104
Dylan16807 wrote: post #97

Earlier quoted context omitted.

I want to work up to 8 drives in a vdev as my storage needs grow, because that's cost-effective. But I don't want to spend 8 drives of money up front, because it's a lot and because drive prices drop over time. Am I not a real use case?

For the first 8 drives, but after that no. BRTFS-raid2 seems to be good for up to 8 drives if that is your use case. But there is no easy upgrade path to that 9th drive. You should not be running less than 3 drives for any data you care about. Fortunately, most people don't have data they care about. Their photos and documents are on cloud providers like google/dropbox/facebook (and dozens of others I don't even know…

> BRTFS-raid2 seems to be good for up to 8 drives if that is your use case. But there is no easy upgrade path to that 9th drive. You should not be running less than 3 drives for any data you care about.

What the hell? Can you please clarify what you mean by "BRTFS-raid2", since that's not a term that has previously existed and BTRFS does not have any functionality remotely similar to the long-obsolete traditional RAID2, and none of the RAID functionality that BTRFS does have has any limitations or safety thresholds for going beyond 8 drives, especially not the RAID1 mode (which is the most plausible thing your raid2 could have been a typo of).

If you have an 8-drive btrfs RAID array, going to 9 drives is as simple as `btrfs device add`, optionally followed by a `btrfs balance start`. Neither of those will reduce the redundancy of any data stored in the array. And I can't see any reason why you're suddenly talking about 3 drives right after talking about 8 or 9 drives.

Re: Examining Btrfs

#108

Post by Dylan16807 »

Dylan16807 wrote: post #106
bluGill wrote: post #103

Earlier quoted context omitted.

All raid. Your odds of a dual fail failure are too high. Note, good raid6 should have x and y stripes with no more than 8 drives in any stripe. I have no idea what they do, so maybe they can restripe somehow on one drive. Hiding the implementation detail is good if they do this right. But I don't know how to tell if any do it right.

If 8 drives is safe with dual parity, then significantly more than 8 drives would be safe with triple parity, right?

I did some calculations, with an estimated 3% annual failure rate per drive.

The numbers shift a bit depending on how much you expect drive failures to correlate and how long you take to replace them.

So I analyzed two example cases, using pretty good math based on a binomial distribution.

The first case is a pessimistic one, where after a drive fails we assume that any other possible failures from the next year will hit before we finish rebuilding. In this case, 8 drives with double parity have a 1/740 chance of array failure each year. If we switch to triple parity, we can have 16 drives and a slight improvement in safety.

The second case is a more optimistic one, where after a drive fails we assume that any other possible failures from the next week will hit before we finish rebuilding. In this case, 8 drives with double parity have a 1 / 93 million chance of array failure each week. If we switch to triple parity, we can have 40 drives and a slight improvement in safety.

I would have done "what if all the drive failures happened at once" but that gives >10% odds of an 8 drive array failing, which seems unreasonably harsh.

Re: Examining Btrfs

#109

Post by db48x »

Dylan16807 wrote: post #94
db48x wrote: post #68

Earlier quoted context omitted.

That’s because the btrfs deduplicator is quite a different animal. You could just as easily run fdupes or something similar. Meanwhile ZFS is doing something much more ambitious. In any case, I don’t think that either of them are worth using in most cases. Deduplication is just too expensive to be worth it.

> You could just as easily run fdupes or something similar. If you only deduplicate whole files and soft links or hard links are suitable for your situation.

That’s not the aspect of the deduplication process that I was referring to.

Btrfs only supports offline deduplication, where you periodically run a process that searches for duplicate blocks and combines them.

Meanwhile ZFS supports online deduplication, where every block written is checked against an index to see if it is a duplicate of some existing block.

That index is what takes up the memory that jcalvinowens complained about. When deduplication is turned on, ZFS keeps that whole index in memory all of the time. Meanwhile btrfs only needs that index to be loaded while the deduplicator is running, so it doesn’t show up most of the time.

Personally I don’t think that either is really worth the cost; the benefit is so small that the cost isn’t worth paying.

Re: Examining Btrfs

#110

Post by culpable_pickle »

jcalvinowens wrote: post #63
detaro wrote: post #52

Earlier quoted context omitted.

> it's antiquated because of the assumption it is possible to get contiguous RAM for that mapping. source on that? (and even if that's the case, wouldn't that be an implementation detail? E.g. implementations can spill it to L2ARC, but somehow not to different memory locations?)

My memory is that this mapping does have to be contiguous, but I'll dig it up in the code and link it here either way.

still planning on linking that code?
Post reply on HN