Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS: Use mirror vdevs, not RAIDZ

jrs-s.net

Post reply on HN
Print view View original

101–110 of 133 posts

Re: ZFS: Use mirror vdevs, not RAIDZ

#101

Post by mindslight »

necheffa wrote: post #24
tjoff wrote: post #22

I don't agree. And if one of your disks failed, and age was a factor… you’re going to be sweating bullets wondering if another will fail before your resilver completes. So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool. I sure do hope you aren't on vacation and/or have to order a drive online. It all depends on your use case. For me, raidz3 wins ea…

Repeat after me: RAID is not a backup solution, its an uptime solution.

This is true in a vacuum, but I feel it misses the mark having run my own personal array for 20+ years. For the most part, individuals have very little mission critical data (unless they're videographers or professional photographers). Ergo, most data on personal arrays is going to be non-critical - eg VM images, old drive images, build trees, downloaded datasets, torrented media.

Does it make sense to double your cost (/halve your capacity) by backing up the whole thing, especially when the backup is likely to be more spinning rust? I've recently chosen to do so (while also backing up the critical data several different ways), but haven't always. I can understand people who still choose otherwise. Either way my main concern is a single raidz3 and I'm going to have a bad week if I lose that, independent of what can be rebuilt.

In general, personal users of ZFS are very different than institutional users of ZFS. Hopefully not too divergent, so those raidz code paths still receive lots of testing and support. But you need to always keep this in mind when reading any advice about ZFS.

Re: ZFS: Use mirror vdevs, not RAIDZ

#102

Post by myrond »

aidenn0 wrote: post #52
rsync wrote: post #37

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable . We all sleep quite well with this arrangement and have…

I was running 3x 12 drive vdevs in raidz2 and write performance was terrible . We moved some data to a different machine and rebuilt as 18 mirrors. This was a long time ago (as in running on Solaris long time ago), so maybe things are better now.

Each 12 drive array would have an interesting physical block layout.

What was your ASHIFT and what was your physical disk raw sector size because if you had a ASHIFT=9 you would have 5120k blocks, if it was a ASHIFT=12 you would have physical 40960 blocks. None of those line up well with zfs.

Additionally if this was a long time ago the pool would have defaulted to ASHIFT=9 and if you added any 4K drives you could have been having write amplification occurring.

Looks as if you had your drives doing a lot of unnecessary extra wasted work.

Re: ZFS: Use mirror vdevs, not RAIDZ

#103

Post by myrond »

_jal wrote: post #31
js2 wrote: post #18

This article is pretty hand-wavy. It doesn't give any empirical numbers at all. I've had a small FreeNAS server using 4 x 3TB SATA drives in a mirrored config for years now and it's out of space. I'm about to build a new server using used 10 x 3TB SAS drives and intend to put all ten disks into a RAIDZ2 vdev. I care more about space than performance or rebuild times. Before I load it with data, I'll do some testing o…

The article is another entry in a long series of bad ZFS articles. For some reason a lot of people get to a point where they're comfortable with it and suddenly their use case is everyone's, they've become an expert, and you should Just Do What They Say. I highly recommend people ignore articles like this. ZFS is very flexible, and can serve a variety of workloads. It also assumes you know what you're doing, and the…

bang-on.

This article should be thrown in the trash.

The EXAMPLES for a raidz and raidz2 and raidz3 are mis-aligned in the article. They will obviously have terrible performance, and waste space due to blocks not fitting nicely due to non-standard physical block sizes.

mirror's are easy because you don't have to worry about ASHIFT values, and physical block sizes of devices and making sure all of your blocks can be divided nicely into 128K block sizes.

I get the feeling that the author of the article never actually has ran a properly configured raidz(x) with SLOG and cache devices.

Re: ZFS: Use mirror vdevs, not RAIDZ

#104

Post by aDfbrtVt »

rsync wrote: post #86
aDfbrtVt wrote: post #70

Earlier quoted context omitted.

I love the backup solution you guys provide with Borg, the pricing is amazing and the product has been rock solid. Any chance of getting similar "expert level" pricing for accounts using ZFS send | receive ?

zfs send enabled accounts are (relatively) expensive because we need to give you a full-blown VPN with a routable ipv4, etc. If you can make do with an ipv6 address and if you have a reasonable quantity, email us ... perhaps we can work something out ...

I’m relatively new to ZFS, but I thought you can send/receive over ssh (much like rsync). What’s the purpose of the VPN layer?

My use is relatively small, just a home user with less than a TB worth of data. Definitely not worthwhile for your team to set up as a one off.

Re: ZFS: Use mirror vdevs, not RAIDZ

#105

Post by 67868018 »

tjoff wrote: post #22

I don't agree. And if one of your disks failed, and age was a factor… you’re going to be sweating bullets wondering if another will fail before your resilver completes. So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool. I sure do hope you aren't on vacation and/or have to order a drive online. It all depends on your use case. For me, raidz3 wins ea…

Your mirrors can be more than 2 wide, you can have automatic hot standbys, you can make each mirror its own zpool so loss of some drives doesn’t lose the entire pool.

Actually losing a mirror vdev doesn’t lose the whole pool anymore for several years now. I’ve recovered data off a zpool where I lost one of three mirrored vdevs. It’s not pretty, but your data is still there. Any files on the missing drives are just 0 bytes.

Re: ZFS: Use mirror vdevs, not RAIDZ

#106

Post by 67868018 »

somehnguy wrote: post #49

No thanks. I use raidz2. With mirror vdevs if you lose the wrong 2 drives you lose everything. I can lose any 2 drives and be totally fine. The probability of losing the wrong 2 drives at once is small, sure. But I would rather just not care about that probability. And I don't lose half my capacity, which for a home user (I don't have an unlimited budget!) matters a whole lot more than having the absolute best iops.

You don’t lose everything, only the files that were on that vdev. I’ve been through this. It’s not fun, though, and your pool is irreversibly damaged but your data is not all lost

Re: ZFS: Use mirror vdevs, not RAIDZ

#107

Post by dnautics »

rsync wrote: post #94
dnautics wrote: post #74

Earlier quoted context omitted.

thank you for this. can you speak to the retention rate post culling using your proofing? And do you have preferred manufacturers? Also, do you bother to do "the trick" where you zfs across heterogeneous SKUs to reduce the risk of use-case-coupled failures?

I don't have real numbers for this but I feel like post-burn-in retention is very high - like 99%. We do not have preferred manufacturers. Data suggests that Hitachis are better than Seagates but I reject this kind of Ford vs. Chevy preference in hard drives - the "good drives" can flip immediately - with new model or even new revision of existing model. I don't think it makes sense to try to pursue a particular manu…

Thanks for the good info!

Re: ZFS: Use mirror vdevs, not RAIDZ

#108

Post by rsync »

silenteh wrote: post #100
rsync wrote: post #37

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable . We all sleep quite well with this arrangement and have…

May I ask if you run a distributed filesystem on top of ZFS and if so which one ?

No. We keep things as simple as possible.

Re: ZFS: Use mirror vdevs, not RAIDZ

#109

Post by mindslight »

Fnoord wrote: post #91
res0nat0r wrote: post #76

Earlier quoted context omitted.

Backing up TB's of data isn't cheap no matter what type of cloud backup solution you're picking unfortunately. I don't think most here want to spend $1000/year on backing up their entire NAS.

> I don't think most here want to spend $1000/year on backing up their entire NAS. These 2 solutions cost less than 10 USD/month: Jottacloud [1] and zfs.rent [2]. The former is even hosted in privacy-friendly Norway, and its 5 TB unlimited with going over that you get upload-capped. So I pay 90 EUR/year for Jottacloud. Which is approx 10% of your projected 1000 USD. [1] https://www.jottacloud.com/en/ [2] https://zfs.…

1. "Unlimited" plans marketed for personal users are unsustainable, which is why they invariably push you into some proprietary app. Sure they might let you dump 24TB on it while they're growing, but they'll inevitably tighten that up. And a key feature of backups is long term - do you really want to be reuploading 24TB in a year or two? Look at their business pricing for the actual no-nonsense cost.

2. You can build a 8TBx4+2 raidz2 with EasyStores for $720. Backing that up to a raidz1 zfs.rent is $1200 for a single year. $2500 for 3 years by prepurchasing the drives.

An estimate of $1000/year for full cloud backups isn't wrong.

Re: ZFS: Use mirror vdevs, not RAIDZ

#110

Post by angry_octet »

Rafuino wrote: post #73
angry_octet wrote: post #27

I would like to see an update for this for SSDs. The devices are black boxes of mystery, possibly likely to fail nearly simultaneously (at least in terms of being unable to write). Early on we left 20% if capacity unformatted to allow more wear leveling, but running them in RAID1 seemed both necessary and likely to cause synchronous errors. RAIDZ1 over three disks might be okay. Obviously huge R/W bandwidth so resilv…

Battery backed flash sounds like a huge PitA to deal with though. Where do you even get that?

The battery is on the HBA. If you lose power it will replay all uncommitted writes. Specifically, to the ZIL but for everything else too. (Originally these were batteries but they are mainly supercapacitors now.)

Just saw this interesting SNIA presentation, which indicates that having NVDIMMs might be even better than NVMe for ZIL.

https://www.snia.org/sites/default/files/SDC/2018/presentati...

NVDIMM as RAM+FLASH+capacitor: https://www.vikingtechnology.com/products/nvdimm/

Post reply on HN