Flavour: Ubuntu Server Headline: Dump ZFS on Linux Affiliation: Containers @ Netflix Description: ZFS On Linux is poorly integrated mess through the SPL. Memory management is an active detractor from performance, stability, and operations. It's unlikely that it's ever going to be "native" on Linux. Even more unlikely is its integration into upstream. Unfortunately, ZoL, unlike other out of tree additions Canonical ha…
> Whoever seems to be singing the praises of ZFS on Linux hasn't put it through its paces in modern, multi-tenant container workloads. I ran a Hadoop Cluster with it? Does that count? Your problem is probably the ARC and memory problems due to slow shrinking or stuff like that? There is some work or at least the intention to use the pagecache infrastructure for the ARC to make things more smooth. However at the momen…
The issues I've hit are the following:
https://github.com/zfsonlinux/zfs/issues/5814
https://github.com/zfsonlinux/zfs/issues/5535
Unfortunately, these problems stem from page faults occurring inside of VM that propagate to ZFS. If the fault occurs during cgroup memory pressure, ZFS / SPL may fail to allocate memory. ZFS will never get out of this case unless memory is freed up elsewhere in the hierarchy.
The other issue we had was with ZFS integration. There are a few symbols around mounts that are exported GPL only, and they interfere with volume GC, and mounting, causing some issues with standard tools like Systemd, and others.
For the most part, our container applications are stateless, or soft state, in this, we don't rely on RAID1, 5, 6, etc... but instead RAID0, or no RAID at all. If we detect a bad disk, we'd rather just evacuate the containers, and restore state later.
I'd love to hear about your experience with BtrFS. What issues have you seen with it (single disk, or RAID0 -- I know RAID1, 5, and 6 are hokey at best)? How did the project handle your issues?