Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ZFS High-Availability NAS

github.com

Post reply on HN
Print view View original

11–20 of 62 posts

Re: ZFS High-Availability NAS

#11

Post by qaq »

insaneirish wrote: post #4

It's very nice to see someone putting in the work to document this and show others how they were able to get it to work. But I unfortunately believe, quite sincerely, that redundancy at this level is misguided at best and dangerous at worst. Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box th…

There are legitimate reasons to have a CP system and not an AP system, it's obviously application specific. Also over-engineered systems come with their own risks and can experience as much down time as a well setup CP system due to human error stemming from the system's complexity (AWS is a good example).

Re: ZFS High-Availability NAS

#12

Post by Veratyr »

To add to the downsides, you can't expand RAIDZ vdevs.

If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev.

Unfortunately I don't think there's an expandable file system that handles parity/erasure coding as well as bitrot that can be used right now. BTRFS might be usable after the RAID5/6 rewrite and Ceph might be more usable after Bluestore comes along (at the moment erasure coding performance seems to be pretty bad, even with an SSD cache) but in the present there's nothing.

I really hope https://www.patreon.com/bcachefs manages to fix this.

Re: ZFS High-Availability NAS

#13

Post by X86BSD »

olavgg wrote: post #3

FreeBSD has a cool HA ZFS solution you can use. HAST https://wiki.freebsd.org/HAST There are also other solutions which you can hack together like iSCSI or GEOM Gate Network https://www.freebsd.org/doc/handbook/geom-ggate.html I would however strongly recommend a proper distributed storage solution like Ceph or GlusterFS.

People will do whatever they want to do. And my comment is not a knock on HAST.

But myself, and I know I am not alone, firmly believe especially regarding HA and ZFS that following anything other than the KISS principle is pain upon pain upon pain.

Wether it's iSCSI, HAST, NFS, HAST or any combination thereof.

The simplest and least painful? Is to simply create two servers with dual nic's and dual HBA's.

Assign half the disks in a mirror to HBA-1 the other half to HBA-2.

Use one nic for public networking, use the other for a direct P2P Gig-E link between server A and server B.

Use zfs replication over the private network link using the granularity you feel most comfortable with. 30 minutes? 15 minutes? 5? It's up to you.

This is the simplest, and IMO most reliable way to go about this. When you start adding in iSCSI, NFS, HAST etc you are simply making things to complex, introducing multiple points of failure, and the pain when something does break? Ill pass. I'd rather have to manually intervene and fail over to server B if server A fails and vice versa than deal with all the pain of additional things that could break or introduce split brained issues etc.

But it's a free world, use whatever solution fits your comfort zone.

Re: ZFS High-Availability NAS

#14

Post by olavgg »

zokier wrote: post #6
cmurf wrote: post #5

Earlier quoted context omitted.

Yeah I think only distributed fs's like Ceph and Gluster really scale. File systems, even ZFS and Btrfs, don't scale because they don't deal with the fact it's not always one or two hard drives dying that'll take out the storage stack. Network card, power supply, motherboard, etc. And unless you have an all SSD array, the recovery times aren't scaling. One dead drive used to take maybe an hour to rebuild and now it c…

Is the performance of Ceph/Gluster comparative to ZFS performance? I would have thought them to be inherently massively slower

It is slower, but still probably good enough for many use cases.

https://software.intel.com/en-us/blogs/2013/10/25/measure-ce...

http://www.intel.com/content/dam/www/public/us/en/documents/...

Re: ZFS High-Availability NAS

#15

Post by guitarbill »

insaneirish wrote: post #4

It's very nice to see someone putting in the work to document this and show others how they were able to get it to work. But I unfortunately believe, quite sincerely, that redundancy at this level is misguided at best and dangerous at worst. Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box th…

> Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box that is assumed to be 100% available. These assumptions about reliablity of systems and their consistent behavior under network partitions has bitten countless people before and will continue to claim victims unless application paradigms are fundamentally rethought.

I'd recommend this... except that it's really, really hard [0]. And if you think about it, you have to do this for every new application, which is a bit wasteful. So IMO, doing it at filesystem level is a good trade off. It's abstract enough to be useful over a wide range of use-cases (unlike block storage), because if you think about it file systems are just frameworks.

To be fair, block can work really well. I think block storage vendors have themselves to blame. Half the "standards" aren't, interop can be terrible. I know I'd rather have simple, reliable storage - stop stuffing "enterprise" features into a rotting, increasingly technical-debt laden codebase. All back-up solutions for block suck - you'd think this problem would be fixed, especially with cloud storage. I could go on.

[0] https://www.somethingsimilar.com/2013/01/14/notes-on-distrib...

Re: ZFS High-Availability NAS

#16

Post by zzzcpan »

qaq wrote: post #11
insaneirish wrote: post #4

It's very nice to see someone putting in the work to document this and show others how they were able to get it to work. But I unfortunately believe, quite sincerely, that redundancy at this level is misguided at best and dangerous at worst. Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box th…

There are legitimate reasons to have a CP system and not an AP system, it's obviously application specific. Also over-engineered systems come with their own risks and can experience as much down time as a well setup CP system due to human error stemming from the system's complexity (AWS is a good example).

NASes are not CP systems, they are noCAP systems. They cannot guarantee neither consistency nor availability in the event of network partition.

Re: ZFS High-Availability NAS

#17

Post by [deleted] »

zzzcpan wrote: post #16
qaq wrote: post #11

Earlier quoted context omitted.

There are legitimate reasons to have a CP system and not an AP system, it's obviously application specific. Also over-engineered systems come with their own risks and can experience as much down time as a well setup CP system due to human error stemming from the system's complexity (AWS is a good example).

NASes are not CP systems, they are noCAP systems. They cannot guarantee neither consistency nor availability in the event of network partition.

[deleted]

Re: ZFS High-Availability NAS

#18

Post by [deleted] »

zzzcpan wrote: post #16
qaq wrote: post #11

Earlier quoted context omitted.

There are legitimate reasons to have a CP system and not an AP system, it's obviously application specific. Also over-engineered systems come with their own risks and can experience as much down time as a well setup CP system due to human error stemming from the system's complexity (AWS is a good example).

NASes are not CP systems, they are noCAP systems. They cannot guarantee neither consistency nor availability in the event of network partition.

[deleted]

Re: ZFS High-Availability NAS

#19

Post by zzzcpan »

zimbatm wrote: post #9
cmurf wrote: post #5

Earlier quoted context omitted.

Yeah I think only distributed fs's like Ceph and Gluster really scale. File systems, even ZFS and Btrfs, don't scale because they don't deal with the fact it's not always one or two hard drives dying that'll take out the storage stack. Network card, power supply, motherboard, etc. And unless you have an all SSD array, the recovery times aren't scaling. One dead drive used to take maybe an hour to rebuild and now it c…

Isn't the rebuilding argument valid on the machine level as well though? Granted if your blocks are redundant it's possible to copy from multiple sources but rebuilding is still limited to the network speed.

On a properly designed distributed system disk-replacement costs are spread across all machines and in time, making performance degradation a non-issue.

Re: ZFS High-Availability NAS

#20

Post by ThatPlayer »

Veratyr wrote: post #12

To add to the downsides, you can't expand RAIDZ vdevs. If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev. Unfortunately I don't think there's an expandable file system that handles parity/erasure coding as well as bitrot that can be used right now. BTRFS might be usable after the RAID5/6 rewrite and Ceph might be mo…

I've been looking at SnapRAID and UnRAID which I believe use their own parity and handles expansions. Mostly because I'm currently using BTRFS RAID6 and might move off it.

I also have high hopes for bcachefs. Currently using bcache on my laptop and it's been great. BTRFS doesn't really have an easy SSD cache like ZFS, and it's a hassle to convert over to a Bcache backed BTRFS setup.

Post reply on HN