Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS: Use mirror vdevs, not RAIDZ

jrs-s.net

Post reply on HN
Print view View original

81–90 of 133 posts

Re: ZFS: Use mirror vdevs, not RAIDZ

#81

Post by unethical_ban »

gh02t wrote: post #53
minimaul wrote: post #13

Earlier quoted context omitted.

The one thing in the article I agree with without reservation though is that you should always have a backup of your pool! ZFS makes doing good backups easy, with zfs send | zfs receive.

I mean, is this controversial? The problem for a budget conscious home user is: where do you back up to? Doing a complete backup to a remote storage provider isn't a practical option for many users, which leaves the affordable options being to only back up critical data and/or use something like a big USB external drive. Neither of these options are great, so in reality many people end up relying on RAID or mirroring…

I have a raidz2 that I keep everything on: (A) Personal, (B) core media, and (C) all other media.

I backup A and B on a workstation drive, and I further backup A to S3 Glacier for $1/month/100GB, syncing every few months.

It isn't all or nothing, 3-copies vs. striped pool. Mirrors are fine if you have quick access to fix dead drives, but I like the idea of being able to lose any two drives vs. only one drive in a given mirror. And while I currently have a 4 drive RAIDz2 (same disk utilization as a mirror) I wish I had a 4+2 array, which I'll probably build next year.

Re: ZFS: Use mirror vdevs, not RAIDZ

#82

Post by zepearl »

minimaul wrote: post #8

As always with RAID-style setups, there’s an inevitable trade off of cost vs capacity vs performance. There’s still a place for RAIDZ/RAIDZ2, and in my opinion that place is storing bulk data that isn’t too heavily accessed or that needs to be stored with an eye towards keeping £/GB down. Yes, mirrors are faster. Yes, mirrors are easier to expand. But across 12 4TB disks that is 24TB instead of 40TB with RAIDZ2 - and…

Btw., I understand that with OpenZFS 2 and its new "sequential resilvering" feature ( https://arstechnica.com/gadgets/2020/12/openzfs-2-0-release-... ) rebuilding should be faster or should at least put less pressure on the drives (less random I/O)... .

Edit: sorry, just noticed that this was already mentioned in another subthread.

Re: ZFS: Use mirror vdevs, not RAIDZ

#83

Post by bewaretheirs »

zepearl wrote: post #82
minimaul wrote: post #8

As always with RAID-style setups, there’s an inevitable trade off of cost vs capacity vs performance. There’s still a place for RAIDZ/RAIDZ2, and in my opinion that place is storing bulk data that isn’t too heavily accessed or that needs to be stored with an eye towards keeping £/GB down. Yes, mirrors are faster. Yes, mirrors are easier to expand. But across 12 4TB disks that is 24TB instead of 40TB with RAIDZ2 - and…

Btw., I understand that with OpenZFS 2 and its new "sequential resilvering" feature ( https://arstechnica.com/gadgets/2020/12/openzfs-2-0-release-... ) rebuilding should be faster or should at least put less pressure on the drives (less random I/O)... . Edit: sorry, just noticed that this was already mentioned in another subthread.

There's also the new "special" vdev feature (which allow you to segregate metadata onto separate ssd mirrors) which should further speed up the remaining unavoidable random reads during scrub/resilver (for metadata traversal).

Re: ZFS: Use mirror vdevs, not RAIDZ

#84

Post by caillou »

jryb wrote: post #10

My gripe with this is that the author assumes everyone has the same workload and prefers the same set of tradeoffs. I use raidz1 on my home desktop precisely because I would absolutely prefer to have to wait for a resilver than lose data. “So backup your data!” - of course, but that’s just an implicitly larger pool.

I use 10x8TB in RaidZ 2 in my home server. TimeMachine Backup for 6 people, docker volumes and an excessively huge media collection.

The TimeMachine datasets are backed up offsite.

Losing this pool would be a PITA, but not critical.

My primary goal with ZFS is some data redundancy. At a good cost. And quick remote backup for a fraction of the pool. Not performance.

At one point, 2 disks died within 2 days. While there was some panic involved, the data on the server could be reproduced with some time.

There isn’t a best solution, that fits all needs. If there was, ZFS wouldn‘t offer all the options it does.

Re: ZFS: Use mirror vdevs, not RAIDZ

#85

Post by jacob019 »

joekrill wrote: post #14

I've been using ZFS for a few years now and it's just been amazing. Storage is so cheap these days that it's just much simpler and straight-forward to use mirrored vdevs. If a single drive fails, the pool is still completely usable, and all I have to do is swap out the bad drive (when I get around to it, no hurry usually) and resilver. Resilvering can take a while, but everything is still completely usable while it's…

That's risky. Better to add the new drive and re-silver before removing the old drive.

Re: ZFS: Use mirror vdevs, not RAIDZ

#86

Post by rsync »

aDfbrtVt wrote: post #70
rsync wrote: post #37

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable . We all sleep quite well with this arrangement and have…

I love the backup solution you guys provide with Borg, the pricing is amazing and the product has been rock solid. Any chance of getting similar "expert level" pricing for accounts using ZFS send | receive ?

zfs send enabled accounts are (relatively) expensive because we need to give you a full-blown VPN with a routable ipv4, etc.

If you can make do with an ipv6 address and if you have a reasonable quantity, email us ... perhaps we can work something out ...

Re: ZFS: Use mirror vdevs, not RAIDZ

#87

Post by djmobley »

bluGill wrote: post #44
necheffa wrote: post #24

Earlier quoted context omitted.

Repeat after me: RAID is not a backup solution, its an uptime solution.

False. There are too many different reasons for a backup to make that statement, and for most people RAID and NAS are combined into one. There is backup because a disks are unreliable and fail randomly. RAID is designed to solve this. There is backup because you accidentally deleted a file. There is backup because ransomware encrypted your files. RAID with snapshots solve these (arguably better than most other backup…

RAID is not a backup.

It really is as simple as that, and it is helpful to remind people so they do not suffer the harms likely to result from mistakenly believing that RAID is a backup.

Re: ZFS: Use mirror vdevs, not RAIDZ

#88

Post by nix23 »

labawi wrote: post #78
nix23 wrote: post #12

Earlier quoted context omitted.

I then tell a real horror-story about my past, not so long ago there was a EVA (A SAN from HP) with a freaking out hardware Raid out of nowhere (uptime about 200 days no Firmware update before nothing) it just started to freak out, the story goes on but the lesson...No more HW-Raid/proprietary stuff for me EVER again.

While none for me either, I used to look down on HW RAID etc. Lately I've realized, such solutions are genuinely useful and more likely to survive/work properly, if the people managing it do not understand the technology, don't want to understand or just don't have the time for it. HW RAID with red blinking lights seems more likely to survive than a SW RAID that works until it doesn't, because you lost all the redund…

>HW RAID with red blinking lights seems more likely to survive than a SW RAID that works until it doesn't, because you lost all the redundancy and noone there knows how either work.

Well if one ever sees your red blinking light because the new printer stands in front of the server :)

Re: ZFS: Use mirror vdevs, not RAIDZ

#89

Post by xoa »

dspillett wrote: post #72
cortesoft wrote: post #30

Earlier quoted context omitted.

Isn’t the backup solution to always get two of whatever you build? A primary raid backed up to a second pool?

You need to do more than just keep a backup of current state, if you want to deal with problems other than complete failure/destruction of the primary. What about accidental deletion and incorrect editing (done too long ago for "undo" to work)?

>What about accidental deletion and incorrect editing (done too long ago for "undo" to work)?

Eh? We're talking ZFS here. Built-in atomic low cost snapshots are part of the core feature set. If we're talking using a pool and backing up another pool it's a reasonable baseline assumption that one would be using either their own or one of the many mature scripts to have regular automatic snapshots and retention schedule. Ie., once every 10 min for a day, once an hour for a week or two, once a week for a year, and monthly thereafter (or whatever fits your use case), along with preservation of any manual custom snapshots. I mean, it's right there, for free.

Re: ZFS: Use mirror vdevs, not RAIDZ

#90

Post by gh02t »

unethical_ban wrote: post #81
gh02t wrote: post #53

Earlier quoted context omitted.

I mean, is this controversial? The problem for a budget conscious home user is: where do you back up to? Doing a complete backup to a remote storage provider isn't a practical option for many users, which leaves the affordable options being to only back up critical data and/or use something like a big USB external drive. Neither of these options are great, so in reality many people end up relying on RAID or mirroring…

I have a raidz2 that I keep everything on: (A) Personal, (B) core media, and (C) all other media. I backup A and B on a workstation drive, and I further backup A to S3 Glacier for $1/month/100GB, syncing every few months. It isn't all or nothing, 3-copies vs. striped pool. Mirrors are fine if you have quick access to fix dead drives, but I like the idea of being able to lose any two drives vs. only one drive in a giv…

Yeah, this is basically what I do as well, I didn't mean to imply it is all or nothing. The practical solution is tiered backup of the stuff you really care about + some local fault tolerance for stuff you don't care about as much.

What I wanted to point out is that of course backing up everything is the solution per the parent comment, but the reality is that costs for punting and mindlessly backing up say 10 TB of data properly are still fairly high. So you pick and choose the important stuff and do the best you can for the rest.

Post reply on HN