Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS: Use mirror vdevs, not RAIDZ

jrs-s.net

Post reply on HN
Print view View original

91–100 of 133 posts

Re: ZFS: Use mirror vdevs, not RAIDZ

#91

Post by Fnoord »

res0nat0r wrote: post #76
dsr_ wrote: post #33

Earlier quoted context omitted.

zfs send. Keep different filesystems for system backup, personal data, media storage, gross hoarding. Consider not backing up the media storage and gross hoarding filesystems. Targets: 1. That NAS off in the corner built from an old desktop. 2. rsync.net and similar. 3. your encrypted cloud backup service of choice, through an adapter. If the value of your data is less than the cost to protect it, you know it is disp…

Backing up TB's of data isn't cheap no matter what type of cloud backup solution you're picking unfortunately. I don't think most here want to spend $1000/year on backing up their entire NAS.

> I don't think most here want to spend $1000/year on backing up their entire NAS.

These 2 solutions cost less than 10 USD/month: Jottacloud [1] and zfs.rent [2]. The former is even hosted in privacy-friendly Norway, and its 5 TB unlimited with going over that you get upload-capped. So I pay 90 EUR/year for Jottacloud. Which is approx 10% of your projected 1000 USD.

[1] https://www.jottacloud.com/en/

[2] https://zfs.rent/

Re: ZFS: Use mirror vdevs, not RAIDZ

#92

Post by Fnoord »

mnd999 wrote: post #58
dsr_ wrote: post #33

Earlier quoted context omitted.

zfs send. Keep different filesystems for system backup, personal data, media storage, gross hoarding. Consider not backing up the media storage and gross hoarding filesystems. Targets: 1. That NAS off in the corner built from an old desktop. 2. rsync.net and similar. 3. your encrypted cloud backup service of choice, through an adapter. If the value of your data is less than the cost to protect it, you know it is disp…

2) and 3) requires a decent broadband upload speed. Home broadband often doesn’t provide this.

Depends where you're from I guess.

Its also just the first backup which takes a (long) while. If you got only 10 mbit upload, then you're able to upload 600 mbit per minute, or 36000 mbit per hour. Which is about 4500 mB/hour (or ~4,4 GB/hour). Which results in 108000 mB/day (or ~105 GB/day). Its about 3/4 of a TB a week.

I'm glad I got 30+ mbit upload with my VDSL2. My first backups took ages (I got 5 TB with Jottacloud). First I thought it was Duplicati being slow with regards to cryptography or being .NET, but nope that ain't it.

Incremental backups during night are not going to hamper the connection though.

Re: ZFS: Use mirror vdevs, not RAIDZ

#93

Post by Fnoord »

bluGill wrote: post #44
necheffa wrote: post #24

Earlier quoted context omitted.

Repeat after me: RAID is not a backup solution, its an uptime solution.

False. There are too many different reasons for a backup to make that statement, and for most people RAID and NAS are combined into one. There is backup because a disks are unreliable and fail randomly. RAID is designed to solve this. There is backup because you accidentally deleted a file. There is backup because ransomware encrypted your files. RAID with snapshots solve these (arguably better than most other backup…

If you got disks from the same brand, and same batch, chances are they will fail at the same time before a resilver.

Re: ZFS: Use mirror vdevs, not RAIDZ

#94

Post by rsync »

dnautics wrote: post #74
rsync wrote: post #46

Earlier quoted context omitted.

Some background ... We have a fairly robust drive-proofing procedure prior to deployment - I think we still use 'badblocks' and really beat them up for 5-7 days. It's not enough to merely not fail - we have to see perfect returns on the SMART diagnostics and zero complaints from FreeBSD. So we're starting with a known-good population of drives. Then we monitor them very closely - again with both SMART and FreeBSD/ZFS…

thank you for this. can you speak to the retention rate post culling using your proofing? And do you have preferred manufacturers? Also, do you bother to do "the trick" where you zfs across heterogeneous SKUs to reduce the risk of use-case-coupled failures?

I don't have real numbers for this but I feel like post-burn-in retention is very high - like 99%.

We do not have preferred manufacturers. Data suggests that Hitachis are better than Seagates but I reject this kind of Ford vs. Chevy preference in hard drives - the "good drives" can flip immediately - with new model or even new revision of existing model. I don't think it makes sense to try to pursue a particular manufacturer.

FWIW, we have bought a ton of seagates over the last four years and they have been fine.

Re: ZFS: Use mirror vdevs, not RAIDZ

#95

Post by segfaultbuserr »

somehnguy wrote: post #49

No thanks. I use raidz2. With mirror vdevs if you lose the wrong 2 drives you lose everything. I can lose any 2 drives and be totally fine. The probability of losing the wrong 2 drives at once is small, sure. But I would rather just not care about that probability. And I don't lose half my capacity, which for a home user (I don't have an unlimited budget!) matters a whole lot more than having the absolute best iops.

The original article already included a reply to this question.

> But wait, why would I want to trade guaranteed two disk failure in RAIDZ2 with only 85.7% survival of two disk failure in a pool of mirrors? Because of the drastically shorter time to resilver, and drastically lower load placed on the pool while doing so. The only disk more heavily loaded than usual during a mirror vdev resilvering is the other disk in the vdev – which might sound bad, but remember that it’s no more heavily loaded than it would’ve been as a RAIDZ member. Each block resilvered on a RAIDZ vdev requires a block to be read from each surviving RAIDZ member; each block written to a resilvering mirror only requires one block to be read from a surviving vdev member. For a six-disk RAIDZ1 vs a six disk pool of mirrors, that’s five times the extra I/O demands required of the surviving disks.

I think it's perfectly okay to disagree. But any comment must contain a counterargument to be useful, for example, one could've argued that load is not an issue in a small array and so on, and I may happily accept the other side of the argument. However, I respectfully criticize that your comment doesn't include any counterargument, it's not useful, please read the article more carefully next time.

Re: ZFS: Use mirror vdevs, not RAIDZ

#96

Post by AnIdiotOnTheNet »

Fnoord wrote: post #91
res0nat0r wrote: post #76

Earlier quoted context omitted.

Backing up TB's of data isn't cheap no matter what type of cloud backup solution you're picking unfortunately. I don't think most here want to spend $1000/year on backing up their entire NAS.

> I don't think most here want to spend $1000/year on backing up their entire NAS. These 2 solutions cost less than 10 USD/month: Jottacloud [1] and zfs.rent [2]. The former is even hosted in privacy-friendly Norway, and its 5 TB unlimited with going over that you get upload-capped. So I pay 90 EUR/year for Jottacloud. Which is approx 10% of your projected 1000 USD. [1] https://www.jottacloud.com/en/ [2] https://zfs.…

What about the initial replication? My NAS is currently sitting at about 8TB used, which even at the maximum 10Mb/s upload of my current ISP (best in the area of course, because 'murika) would take 75 days. Your first link doesn't seem to say anything about alternatives for getting the initial data set to them, and your second link, well "We have queued up 200 users and have decided to close new requests for the time being. Please check back soon!" basically means it is imaginary for the time being.

Re: ZFS: Use mirror vdevs, not RAIDZ

#97

Post by AnIdiotOnTheNet »

Fnoord wrote: post #92
mnd999 wrote: post #58

Earlier quoted context omitted.

2) and 3) requires a decent broadband upload speed. Home broadband often doesn’t provide this.

Depends where you're from I guess. Its also just the first backup which takes a (long) while. If you got only 10 mbit upload, then you're able to upload 600 mbit per minute, or 36000 mbit per hour. Which is about 4500 mB/hour (or ~4,4 GB/hour). Which results in 108000 mB/day (or ~105 GB/day). Its about 3/4 of a TB a week. I'm glad I got 30+ mbit upload with my VDSL2. My first backups took ages (I got 5 TB with Jottac…

> Its about 3/4 of a TB a week.

That's best case. Anyone who's done large transfers over the internet knows reality will be much worse.

Re: ZFS: Use mirror vdevs, not RAIDZ

#98

Post by [deleted] »

rsync wrote: post #37

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable . We all sleep quite well with this arrangement and have…

[deleted]

Re: ZFS: Use mirror vdevs, not RAIDZ

#99

Post by segfaultbuserr »

aidenn0 wrote: post #52
rsync wrote: post #37

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable . We all sleep quite well with this arrangement and have…

I was running 3x 12 drive vdevs in raidz2 and write performance was terrible . We moved some data to a different machine and rebuilt as 18 mirrors. This was a long time ago (as in running on Solaris long time ago), so maybe things are better now.

Slow performance is always my impression with RAIDZ. There's an obvious performance hit if you are coming from conventional RAID setups (like mdadm). I've even seen some unbelievable slow speed like 1 MiB/s in the middle of copying a large git repo (which have a lot of small files, repeated git pull also creates a lot of fragmentation). But my experience was based on early ZFSOnLinux, perhaps a real BSD will perform better. Anyway, I knew ZFS's first goal is data safety, not performance, and it's why I could tolerate the performance...

Re: ZFS: Use mirror vdevs, not RAIDZ

#100

Post by silenteh »

rsync wrote: post #37

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable . We all sleep quite well with this arrangement and have…

May I ask if you run a distributed filesystem on top of ZFS and if so which one ?
Post reply on HN