Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS: Use mirror vdevs, not RAIDZ

jrs-s.net

Post reply on HN
Print view View original

121–130 of 133 posts

Re: ZFS: Use mirror vdevs, not RAIDZ

#121

Post by Fnoord »

mindslight wrote: post #109
Fnoord wrote: post #91

Earlier quoted context omitted.

> I don't think most here want to spend $1000/year on backing up their entire NAS. These 2 solutions cost less than 10 USD/month: Jottacloud [1] and zfs.rent [2]. The former is even hosted in privacy-friendly Norway, and its 5 TB unlimited with going over that you get upload-capped. So I pay 90 EUR/year for Jottacloud. Which is approx 10% of your projected 1000 USD. [1] https://www.jottacloud.com/en/ [2] https://zfs.…

1. "Unlimited" plans marketed for personal users are unsustainable, which is why they invariably push you into some proprietary app. Sure they might let you dump 24TB on it while they're growing, but they'll inevitably tighten that up. And a key feature of backups is long term - do you really want to be reuploading 24TB in a year or two? Look at their business pricing for the actual no-nonsense cost. 2. You can build…

It isn't "unlimited", it is "unlimited with caveat". It is 5 TB soft cap after which upload limits apply, which get progressively harsh the more you are from that 5 TB soft limit. Just like I wrote in my previous post, and as they write on their website. For me, 5 TB in a privacy-friendly country is a bargain deal. Duplicati has support for Jottacloud, and they also provide an CLI for their API.

If I had an amount of 24 TB to back up I'd buy my own server and harddisks, and I'd physically set that server up at work or at home. Then, travel to the data center (I live in Amsterdam Area, so enough choice) and put the server there, again, physically. No need to "upload 24 TB".

RAIDZ et al. is about availability. Which makes sense for businesses, but for personal data an offsite backup makes more sense. I'd just go with a mirror of 2 different disks, with different serial numbers (ie. different orders).

Re: ZFS: Use mirror vdevs, not RAIDZ

#122

Post by Fnoord »

AnIdiotOnTheNet wrote: post #96
Fnoord wrote: post #91

Earlier quoted context omitted.

> I don't think most here want to spend $1000/year on backing up their entire NAS. These 2 solutions cost less than 10 USD/month: Jottacloud [1] and zfs.rent [2]. The former is even hosted in privacy-friendly Norway, and its 5 TB unlimited with going over that you get upload-capped. So I pay 90 EUR/year for Jottacloud. Which is approx 10% of your projected 1000 USD. [1] https://www.jottacloud.com/en/ [2] https://zfs.…

What about the initial replication? My NAS is currently sitting at about 8TB used, which even at the maximum 10Mb/s upload of my current ISP (best in the area of course, because 'murika) would take 75 days . Your first link doesn't seem to say anything about alternatives for getting the initial data set to them, and your second link, well "We have queued up 200 users and have decided to close new requests for the tim…

Buy a second hand "enterprise" server and buy some new HDDs. Set the server up at work/home, and physically put the server at a data center near your vicinity. If you already live in USA, hosting elsewhere doesn't get you much. The problem with hosting in USA is for sensitive data for people outside of USA (EU for example).

Re: ZFS: Use mirror vdevs, not RAIDZ

#123

Post by seized »

aidenn0 wrote: post #52
rsync wrote: post #37

We (rsync.net) have several PB of raidz3 deployed all over the world. We use conservatively sized (12-15 drive) vdevs and typically join 3 or 4 of those together to make a pool. I can see getting nervous about raidz2 (sort of analogous to "raid6") after a drive failure ... but losing 4 drives out of 12 in a single raidz3 failure cascade is extremely improbable . We all sleep quite well with this arrangement and have…

I was running 3x 12 drive vdevs in raidz2 and write performance was terrible . We moved some data to a different machine and rebuilt as 18 mirrors. This was a long time ago (as in running on Solaris long time ago), so maybe things are better now.

RAIDZ gives roughly the IOPS of one drive per VDEV. Mirrors is one drive of write IOPS per VDEV and some say two drives of IOPS for read.

You went from 3 drives to 18, IOPS wise.

Re: ZFS: Use mirror vdevs, not RAIDZ

#124

Post by mindslight »

Fnoord wrote: post #121
mindslight wrote: post #109

Earlier quoted context omitted.

1. "Unlimited" plans marketed for personal users are unsustainable, which is why they invariably push you into some proprietary app. Sure they might let you dump 24TB on it while they're growing, but they'll inevitably tighten that up. And a key feature of backups is long term - do you really want to be reuploading 24TB in a year or two? Look at their business pricing for the actual no-nonsense cost. 2. You can build…

It isn't "unlimited", it is "unlimited with caveat". It is 5 TB soft cap after which upload limits apply, which get progressively harsh the more you are from that 5 TB soft limit. Just like I wrote in my previous post, and as they write on their website. For me, 5 TB in a privacy-friendly country is a bargain deal. Duplicati has support for Jottacloud, and they also provide an CLI for their API. If I had an amount of…

I was speaking generally. I'm glad the service fits for you, and it appears that they have setup reasonably-structured incentives.

But using that pricing, 24TB with Jottacloud would be 450 EUR/year, which is still on the order of $1000. The larger point is that disk sizes are always growing, and while 24TB may seem unnecessary to you, it's very easy to build an array that is cumbersome to backup. And IME this has remained true for the past two decades.

You're right that the cost effective way of performing large backups seems to be just buying another array and hosting it somewhere offsite. But that still doubles your array cost, and so is not something to be done lightly.

FWIW personal users want availability too, unless rebuilding arrays from scratch is your hobby. A simple mirror is a form a raid, which you can only get away with if your storage requirements can be taken care of by a single spindle. If that's where you are, fantastic. But that doesn't describe most home ZFS users.

Re: ZFS: Use mirror vdevs, not RAIDZ

#125

Post by mindslight »

denkmoon wrote: post #112
res0nat0r wrote: post #76

Earlier quoted context omitted.

Backing up TB's of data isn't cheap no matter what type of cloud backup solution you're picking unfortunately. I don't think most here want to spend $1000/year on backing up their entire NAS.

b2 backblaze is quite cheap.

16TB is basically the smallest array where someone would face the question of mirrors vs raidz. 16TB * $0.005 / GB-mo = $960/year.

Re: ZFS: Use mirror vdevs, not RAIDZ

#126

Post by Fnoord »

justinclift wrote: post #119
Fnoord wrote: post #92

Earlier quoted context omitted.

Depends where you're from I guess. Its also just the first backup which takes a (long) while. If you got only 10 mbit upload, then you're able to upload 600 mbit per minute, or 36000 mbit per hour. Which is about 4500 mB/hour (or ~4,4 GB/hour). Which results in 108000 mB/day (or ~105 GB/day). Its about 3/4 of a TB a week. I'm glad I got 30+ mbit upload with my VDSL2. My first backups took ages (I got 5 TB with Jottac…

Bearing in mind that it's likely not practical to use 100% of available upload bandwidth for extremely long periods of time. Generally speaking anyway. :)

True, you're both right, thanks. I should've added that as a disclaimer.

Re: ZFS: Use mirror vdevs, not RAIDZ

#127

Post by Dylan16807 »

mulmen wrote: post #117
ubercow13 wrote: post #79

Earlier quoted context omitted.

They are extremely expensive.

Wow yeah I just did the math again and they are! Tarnsap is 10x S3 prices? I know the service does a lot but I thought it was a small margin on top of plain object storage.

It's hard to price a backup service per byte when all the value comes from software. At the small end there's barely any profit, and at the big end the costs get ridiculous.

Re: ZFS: Use mirror vdevs, not RAIDZ

#128

Post by Dylan16807 »

mindslight wrote: post #124
Fnoord wrote: post #121

Earlier quoted context omitted.

It isn't "unlimited", it is "unlimited with caveat". It is 5 TB soft cap after which upload limits apply, which get progressively harsh the more you are from that 5 TB soft limit. Just like I wrote in my previous post, and as they write on their website. For me, 5 TB in a privacy-friendly country is a bargain deal. Duplicati has support for Jottacloud, and they also provide an CLI for their API. If I had an amount of…

I was speaking generally. I'm glad the service fits for you, and it appears that they have setup reasonably-structured incentives. But using that pricing, 24TB with Jottacloud would be 450 EUR/year, which is still on the order of $1000. The larger point is that disk sizes are always growing, and while 24TB may seem unnecessary to you, it's very easy to build an array that is cumbersome to backup. And IME this has rem…

Unless you have tons of data churn or really love burst uploads, you should treat Jottacloud as roughly 15TB of capacity. Here's the speed chart: https://docs.jottacloud.com/en/articles/3271114-reduced-uplo...

So for 24TB you would use two accounts, assuming they're fine with that, and your uploads would run at 5Mbps per thread, with 12 threads in parallel.

That will let you upload 100GB every 4 hours, which should be enough. And the total cost would be 180EUR/year, which is pretty far from $1000.

Re: ZFS: Use mirror vdevs, not RAIDZ

#129

Post by mulmen »

Dylan16807 wrote: post #127
mulmen wrote: post #117

Earlier quoted context omitted.

Wow yeah I just did the math again and they are! Tarnsap is 10x S3 prices? I know the service does a lot but I thought it was a small margin on top of plain object storage.

It's hard to price a backup service per byte when all the value comes from software. At the small end there's barely any profit, and at the big end the costs get ridiculous.

Yeah I don’t begin to understand the economics. I was just surprised at the difference.

I naively thought storage is cheap so backups are too.

Re: ZFS: Use mirror vdevs, not RAIDZ

#130

Post by jjav »

tjoff wrote: post #22

I don't agree. And if one of your disks failed, and age was a factor… you’re going to be sweating bullets wondering if another will fail before your resilver completes. So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool. I sure do hope you aren't on vacation and/or have to order a drive online. It all depends on your use case. For me, raidz3 wins ea…

> So every every single time you loose any drive in a mirror setup you risk all the data on all drives in the entire pool.

Not true. If you lost all the drives in a mirror vdev, you lost those files not the entire pool.

More importantly, mirrors are not limited to two drives. You can have any number.

I use mirror vdevs with 4 drives.

Post reply on HN