ZFS High-Availability NAS
github.com
ZFS High-Availability NAS
1–10 of 62 posts
Re: ZFS High-Availability NAS
#2I think this is quite amazing, building high-availability storage is a complex endeavour because the entire storage path needs to be 100% redundant.
It's quite an achievement to get this functional using open-source.
Re: ZFS High-Availability NAS
#3FreeBSD has a cool HA ZFS solution you can use.
HAST https://wiki.freebsd.org/HAST
There are also other solutions which you can hack together like iSCSI or GEOM Gate Network https://www.freebsd.org/doc/handbook/geom-ggate.html
I would however strongly recommend a proper distributed storage solution like Ceph or GlusterFS.
Re: ZFS High-Availability NAS
#4It's very nice to see someone putting in the work to document this and show others how they were able to get it to work.
But I unfortunately believe, quite sincerely, that redundancy at this level is misguided at best and dangerous at worst.
Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box that is assumed to be 100% available. These assumptions about reliablity of systems and their consistent behavior under network partitions has bitten countless people before and will continue to claim victims unless application paradigms are fundamentally rethought.
Many people already know this. Even people who build these systems know this, but too many applications already exist that can't be change. Nonetheless, it's important for people building such systems for the first time to be aware that they can make a choice early in the design process and that the status quo is not a good choice. In other words, "Every day, somebody's born who's never seen The Flintstones." (https://mobile.twitter.com/hotdogsladies/status/760580954532...)
Re: ZFS High-Availability NAS
#5FreeBSD has a cool HA ZFS solution you can use. HAST https://wiki.freebsd.org/HAST There are also other solutions which you can hack together like iSCSI or GEOM Gate Network https://www.freebsd.org/doc/handbook/geom-ggate.html I would however strongly recommend a proper distributed storage solution like Ceph or GlusterFS.
Yeah I think only distributed fs's like Ceph and Gluster really scale. File systems, even ZFS and Btrfs, don't scale because they don't deal with the fact it's not always one or two hard drives dying that'll take out the storage stack. Network card, power supply, motherboard, etc. And unless you have an all SSD array, the recovery times aren't scaling. One dead drive used to take maybe an hour to rebuild and now it can be a day due to capacity growing faster than performance.
Re: ZFS High-Availability NAS
#6FreeBSD has a cool HA ZFS solution you can use. HAST https://wiki.freebsd.org/HAST There are also other solutions which you can hack together like iSCSI or GEOM Gate Network https://www.freebsd.org/doc/handbook/geom-ggate.html I would however strongly recommend a proper distributed storage solution like Ceph or GlusterFS.
Yeah I think only distributed fs's like Ceph and Gluster really scale. File systems, even ZFS and Btrfs, don't scale because they don't deal with the fact it's not always one or two hard drives dying that'll take out the storage stack. Network card, power supply, motherboard, etc. And unless you have an all SSD array, the recovery times aren't scaling. One dead drive used to take maybe an hour to rebuild and now it c…
Is the performance of Ceph/Gluster comparative to ZFS performance? I would have thought them to be inherently massively slower
Re: ZFS High-Availability NAS
#7Am I reading the output right, that he is creating three-way stripe over raidz1 volumes? Seems like bit odd configuration for HA solution especially. I mean, doesn't that mean that the array can't handle two disk failures except if it gets lucky?
Re: ZFS High-Availability NAS
#8It's very nice to see someone putting in the work to document this and show others how they were able to get it to work. But I unfortunately believe, quite sincerely, that redundancy at this level is misguided at best and dangerous at worst. Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box th…
Given the target use case of this kind of system, you control what you can control, which is the infra, because the software will overwhelmingly be industry specific COTS stuff that realistically wont be changed.
Further its likely that the types of internal apps running could handle a recovery period, makes total loss less of an issue...this is "old school" operations - having DR plans, backups and recovery time targets
Re: ZFS High-Availability NAS
#9FreeBSD has a cool HA ZFS solution you can use. HAST https://wiki.freebsd.org/HAST There are also other solutions which you can hack together like iSCSI or GEOM Gate Network https://www.freebsd.org/doc/handbook/geom-ggate.html I would however strongly recommend a proper distributed storage solution like Ceph or GlusterFS.
Yeah I think only distributed fs's like Ceph and Gluster really scale. File systems, even ZFS and Btrfs, don't scale because they don't deal with the fact it's not always one or two hard drives dying that'll take out the storage stack. Network card, power supply, motherboard, etc. And unless you have an all SSD array, the recovery times aren't scaling. One dead drive used to take maybe an hour to rebuild and now it c…
Isn't the rebuilding argument valid on the machine level as well though? Granted if your blocks are redundant it's possible to copy from multiple sources but rebuilding is still limited to the network speed.
Re: ZFS High-Availability NAS
#10Ed White, the guy behind the Github repo, is a very experienced sysadmin (and the guy who knocked me off the top scoring slot on Server Fault). His experience supporting both COTS and custom applications in a variety of environments is top-notch.