ZFS forces you to choose a drive size a priori and live with it for the life of the system. Thanks but no thanks. I'll stick with OpenStack Swift where I can leverage new larger disks and have my uncoupled highly available storage to boot.
ZFS High-Availability NAS
31–40 of 62 posts
Re: ZFS High-Availability NAS
#32ZFS forces you to choose a drive size a priori and live with it for the life of the system. Thanks but no thanks. I'll stick with OpenStack Swift where I can leverage new larger disks and have my uncoupled highly available storage to boot.
This is not true. Read up on mirrored vdevs. http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-...
Re: ZFS High-Availability NAS
#33It's very nice to see someone putting in the work to document this and show others how they were able to get it to work. But I unfortunately believe, quite sincerely, that redundancy at this level is misguided at best and dangerous at worst. Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box th…
This is about a completely different type of VM application, and a completely different kind of storage architecture. This is about everything being on the storage box, not just any application state. The two nodes export NFS shares, which host the root filesystems of the nodes being run on the VM servers. Most likely, the two storage heads will be existing on the same subnet as the VMWare boxes, thus making a network partition that affects both nodes much, much less likely. Additionally, by having the sort of architecture being presented, you can do things like live migrations of VMs, which is useful when the underlying OS needs to be brought down for maintenance.
There are trade-offs with any architectural decisions, and your idea provides a much more byzantine, complex, and expensive system for little to no trade-off at the scale presented. There are quite a few places where they just need a few servers, and going to your idea just makes things more expensive and more complicated.
Re: ZFS High-Availability NAS
#34It's very nice to see someone putting in the work to document this and show others how they were able to get it to work. But I unfortunately believe, quite sincerely, that redundancy at this level is misguided at best and dangerous at worst. Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box th…
This is about a completely different type of VM application, and a completely different kind of storage architecture. This is about everything being on the storage box, not just any application state. The two nodes export NFS shares, which host the root filesystems of the nodes being run on the VM servers. Most likely, the two storage heads will be existing on the same subnet as the VMWare boxes, thus making a networ…
> This is about a completely different type of VM application, and a completely different kind of storage architecture.
I understand it completely, and I vehemently disagree with everything about it. Providing networked block storage for virtual disks on top of NFS is a recipe for disaster and performance degradation. Just because "everyone" does it doesn't make me like it any more. Using redundant local disk (preferably with ZFS) gets around this problem. Blocks are actually blocks and failures and performance issues are confined to one node.
> Additionally, by having the sort of architecture being presented, you can do things like live migrations of VMs, which is useful when the underlying OS needs to be brought down for maintenance.
I find live migration to be a hack. Once again, solving the wrong problem at the wrong level.
> There are quite a few places where they just need a few servers, and going to your idea just makes things more expensive and more complicated.
Local disk with appropriate backups and application level failover makes a lot more sense and is fundamentally less complicated.
Re: ZFS High-Availability NAS
#35One day we will hopefully have first class ZFS mirroring using shared nothing hardware. Until then dual-pathed JBODs will have to do.
Re: ZFS High-Availability NAS
#36Earlier quoted context omitted.
Mdadm (Linux's built-in software RAID) can expand a RAID array online, and has been able to do so for about a decade. It's not fundamentally that complex of an operation, but there are enough details to be careful of that I don't see why it makes sense to reimplement it in every filesystem.
It's the checksumming, striping, and parity, especially in context of CoW snapshots that complicate this operation.
LVM2 is a filesystem-agnostic layer that gives you CoW snapshots on top of mdadm's striping/parity. From your list, that just leaves checksumming for bitrot, which you can do at the RAID level via mdadm check, or at the filesystem level with sha1sum --tag / sha1sum -c (although maybe that's not as nice as ZFS/btrfs).
Re: ZFS High-Availability NAS
#37Always on the prowl for cheap just a bunch of disks enclosures for my Solaris / SmartOS systems, from the links in the github readme.md, I learned a whole bunch of stuff, for example, that Sun made the J4400 / J4100 just a bunch of disks external serial attached small computer system interface arrays, and these are dirt cheap on ebay:
https://docs.oracle.com/cd/E19928-01/820-3223-14/J4200_J4400...
that in turn led me on the search for the actual host bus adapter supporting this, and behold:
https://docs.oracle.com/cd/E19928-01/820-3222-19/J4200_J4400...
further "digging" on ebay uncovered that these low profile host bus adapters can be had for as cheap as $18.95 USD plus shipping.
Since this stuff is professional grade equipment, the repossession companies are having a really, really hard time reselling this equipment: enterprises will not buy this because it is so specialized that they do not know about it (and they would much rather buy ultra expensive EMC with a support contract), and private persons will not buy this because most people do not even know what it is, or that it exists, never mind that it requires one to have a rack with professionally installed power and possibly cooling.
As a consequence, this equipment goes for peanuts on ebay, and is a way for someone who knows what they are doing to implement highly reliable, enterprise storage solutions for a mere, tiny fraction of the cost.
Not bad for five minutes worth of research, that was time well invested.
Re: ZFS High-Availability NAS
#38Am I reading the output right, that he is creating three-way stripe over raidz1 volumes? Seems like bit odd configuration for HA solution especially. I mean, doesn't that mean that the array can't handle two disk failures except if it gets lucky?
It means that the vdev on top of the three RAIDZ vdevs can lose up to three devices simultaneously, one per RAIDZ each.
Any further failure would result in a complete data loss. Also, if any of the RAIDZ stripes loses more than one vdev simultaneously, this will lead to a complete data loss as well.
Re: ZFS High-Availability NAS
#39ZFS forces you to choose a drive size a priori and live with it for the life of the system. Thanks but no thanks. I'll stick with OpenStack Swift where I can leverage new larger disks and have my uncoupled highly available storage to boot.
That is not correct: one can easily increase the size of the pool by adding larger devices. As soon as the last device is replaced, the pool instantaneously possesses the upgraded capacity. No complicated grow commands, no filesystem expansions. It JustWorks(SM).
zpool set autoreplace=on pool_name
zpool set autoexpand=on pool_name
What cannot be done is a reduction in pool's capacity, but that does not come into these tales.
There is no "apriori size determination", as zpool(1M) uses the entire disks or logical units, and ZFS filesystems use only as much capacity as they require, and no more, leading to extremely efficient disk space utilization.
Re: ZFS High-Availability NAS
#40Earlier quoted context omitted.
It's the checksumming, striping, and parity, especially in context of CoW snapshots that complicate this operation.
LVM2 is a filesystem-agnostic layer that gives you CoW snapshots on top of mdadm's striping/parity. From your list, that just leaves checksumming for bitrot, which you can do at the RAID level via mdadm check, or at the filesystem level with sha1sum --tag / sha1sum -c (although maybe that's not as nice as ZFS/btrfs).
RAID controllers cannot be relied upon to do check sums, as most RAID controllers do not possess this capability, and of those which do, their firmware is opaque and buggy, and therefore cannot be trusted.
If you care about your data, never use hardware RAID, and always use ZFS or Oracle ASM.
mdadm(8) is a good start on GNU/Linux if one is forced to run on an (older) operating system version with only ext3 or XFS, however while it provides administration consistency and scales to many systems, it does not provide data checksumming, and therefore no data corruption detection and no self healing capabilities like Oracle ASM or like (Open)ZFS.