Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS High-Availability NAS

github.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: ZFS High-Availability NAS

#41

Post by Annatar »

X86BSD wrote: post #13
olavgg wrote: post #3

FreeBSD has a cool HA ZFS solution you can use. HAST https://wiki.freebsd.org/HAST There are also other solutions which you can hack together like iSCSI or GEOM Gate Network https://www.freebsd.org/doc/handbook/geom-ggate.html I would however strongly recommend a proper distributed storage solution like Ceph or GlusterFS.

People will do whatever they want to do. And my comment is not a knock on HAST. But myself, and I know I am not alone, firmly believe especially regarding HA and ZFS that following anything other than the KISS principle is pain upon pain upon pain. Wether it's iSCSI, HAST, NFS, HAST or any combination thereof. The simplest and least painful? Is to simply create two servers with dual nic's and dual HBA's. Assign half…

But myself, and I know I am not alone, firmly believe especially regarding HA and ZFS that following anything other than the KISS principle is pain upon pain upon pain.

Rarely have truer words been written. Highly available, highly reliable storage means keeping it as simple as possible, but no simpler. I wonder how long will it take before computer professionals at large realize this?

Re: ZFS High-Availability NAS

#42

Post by ewwhite »

Annatar wrote: post #37

Always on the prowl for cheap just a bunch of disks enclosures for my Solaris / SmartOS systems, from the links in the github readme.md, I learned a whole bunch of stuff, for example, that Sun made the J4400 / J4100 just a bunch of disks external serial attached small computer system interface arrays, and these are dirt cheap on ebay: https://docs.oracle.com/cd/E19928-01/820-3223-14/J4200_J4400... that in turn led me…

This is key. I mean, I focus on the HP side and used ProLiant servers and storage enclosures in my examples... But the Sun/Oracle enclosures and components are very inexpensive off-lease and come at a price-point where it's totally possible to self-support.

Re: ZFS High-Availability NAS

#43

Post by ewwhite »

Annatar wrote: post #38
zokier wrote: post #7

Am I reading the output right, that he is creating three-way stripe over raidz1 volumes? Seems like bit odd configuration for HA solution especially. I mean, doesn't that mean that the array can't handle two disk failures except if it gets lucky?

It means that the vdev on top of the three RAIDZ vdevs can lose up to three devices simultaneously, one per RAIDZ each. Any further failure would result in a complete data loss. Also, if any of the RAIDZ stripes loses more than one vdev simultaneously, this will lead to a complete data loss as well.

This is a simple example setup in a 12-drive enclosure. 9 disks, a global hot-spare disk and two cache drives.

Previously, I would have used straight ZFS mirrors for performance and expandability (raidz VDEVs can't be expanded), but at this scale, the 9-disk 3x3 raidz1 setup consistently outperformed a 10-disk RAID mirror setup in random and sequential I/O.

See: http://i.imgur.com/SeSlmH8.png

Re: ZFS High-Availability NAS

#44

Post by eatstoomuchjam »

ewwhite wrote: post #43
Annatar wrote: post #38

Earlier quoted context omitted.

It means that the vdev on top of the three RAIDZ vdevs can lose up to three devices simultaneously, one per RAIDZ each. Any further failure would result in a complete data loss. Also, if any of the RAIDZ stripes loses more than one vdev simultaneously, this will lead to a complete data loss as well.

This is a simple example setup in a 12-drive enclosure. 9 disks, a global hot-spare disk and two cache drives. Previously, I would have used straight ZFS mirrors for performance and expandability (raidz VDEVs can't be expanded), but at this scale, the 9-disk 3x3 raidz1 setup consistently outperformed a 10-disk RAID mirror setup in random and sequential I/O. See: http://i.imgur.com/SeSlmH8.png

A set of 5 stripes of mirrors has the same problem that the comment brought up - losing the _wrong_ two disks loses all of your data.

In this case, going to raidz2 across all 12 disks would give an extra disk worth of capacity/performance with the ability to lose any 2 disks without data loss and going to raidz3 should give the same capacity/performance with the ability to lose any 3 disks without data loss.

Re: ZFS High-Availability NAS

#45

Post by MikusR »

Veratyr wrote: post #12

To add to the downsides, you can't expand RAIDZ vdevs. If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev. Unfortunately I don't think there's an expandable file system that handles parity/erasure coding as well as bitrot that can be used right now. BTRFS might be usable after the RAID5/6 rewrite and Ceph might be mo…

REFS on top of Storage Spaces

Re: ZFS High-Availability NAS

#46

Post by ewwhite »

eatstoomuchjam wrote: post #44
ewwhite wrote: post #43

Earlier quoted context omitted.

This is a simple example setup in a 12-drive enclosure. 9 disks, a global hot-spare disk and two cache drives. Previously, I would have used straight ZFS mirrors for performance and expandability (raidz VDEVs can't be expanded), but at this scale, the 9-disk 3x3 raidz1 setup consistently outperformed a 10-disk RAID mirror setup in random and sequential I/O. See: http://i.imgur.com/SeSlmH8.png

A set of 5 stripes of mirrors has the same problem that the comment brought up - losing the _wrong_ two disks loses all of your data. In this case, going to raidz2 across all 12 disks would give an extra disk worth of capacity/performance with the ability to lose any 2 disks without data loss and going to raidz3 should give the same capacity/performance with the ability to lose any 3 disks without data loss.

Hot spares and monitoring. RaidZ2 across that number of drives is a performance nightmare and against ZFS best practices.

Re: ZFS High-Availability NAS

#47

Post by tmd83 »

Veratyr wrote: post #12

To add to the downsides, you can't expand RAIDZ vdevs. If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev. Unfortunately I don't think there's an expandable file system that handles parity/erasure coding as well as bitrot that can be used right now. BTRFS might be usable after the RAID5/6 rewrite and Ceph might be mo…

Yeah. I have read about zfs to some degree before but somehow missed this until a couple of days ago. For all its flexibility this looks a very bad limitation. I think this and rebuild time (again related to vdev width) makes zfs very easy to mess up.

The goal of bachefs seem to very lofty though there isn't much detail on the implication of raid design. It seemed to suggest it solves the zfs fragmentation problem, gives better r/w performance with a simpler codebase. Anyone who understand the programmers guide well enough to comment on the viability of the architecture? Does it really solves all the problem of a non-distributed huge filesystem capable of working from a single disk to a big array?

Re: ZFS High-Availability NAS

#48

Post by gsmethells »

chad_c wrote: post #32
gsmethells wrote: post #31

ZFS forces you to choose a drive size a priori and live with it for the life of the system. Thanks but no thanks. I'll stick with OpenStack Swift where I can leverage new larger disks and have my uncoupled highly available storage to boot.

This is not true. Read up on mirrored vdevs. http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-...

50% storage efficiency and 87.5% survival is not an option for us. We store medical images by the petabyte and drives fail much too fast at scale.

Re: ZFS High-Availability NAS

#49

Post by Veratyr »

ymse wrote: post #22
Veratyr wrote: post #12

To add to the downsides, you can't expand RAIDZ vdevs. If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev. Unfortunately I don't think there's an expandable file system that handles parity/erasure coding as well as bitrot that can be used right now. BTRFS might be usable after the RAID5/6 rewrite and Ceph might be mo…

> Ceph might be more usable after Bluestore comes along (at the moment erasure coding performance seems to be pretty bad, even with an SSD cache) I just saturated a 10Gb link in a k=4,m=2 EC configuration on Ceph (Haswell). Too lazy to set up concurrent clients. What do you mean by pretty bad? This is a Hammer cluster without SSD journals or cache. Bluestore was recently released with Jewel, but shouldn't affect EC p…

Maybe I'm doing something horribly wrong. My setup was:

- 6x4TB HDD

- 2x120GB SSD

- OSD per disk

- k=4,m=2 erasure coded HDD pool

- 1 copy replicated (not) SSD pool

- SSD pool writeback cache for HDD pool

I was consistently seeing the SSD writeback cache fill up then the disks thrashed at 100% IO utilization, limiting incoming writes to ~40MB/s from memory.

Re: ZFS High-Availability NAS

#50

Post by gsmethells »

Annatar wrote: post #39
gsmethells wrote: post #31

ZFS forces you to choose a drive size a priori and live with it for the life of the system. Thanks but no thanks. I'll stick with OpenStack Swift where I can leverage new larger disks and have my uncoupled highly available storage to boot.

That is not correct: one can easily increase the size of the pool by adding larger devices. As soon as the last device is replaced, the pool instantaneously possesses the upgraded capacity. No complicated grow commands, no filesystem expansions. It JustWorks(SM). zpool set autoreplace=on pool_name zpool set autoexpand=on pool_name properties must be set on the pool for this to work, either before or after the pool up…

That is a lot of work compared to just adding a disk and rebalancing. No need to replace every disk in a vdev just to increase disk utilization to the entire disk when using Swift. No fear of losing a pool when losing a vdev.
Post reply on HN