Linus: Don't Use ZFS • Hacker News Board

Re: Linus: Don't Use ZFS

#511

Post by bcantrill » Fri, Jan 10, 2020, 9:37 PM UTC

notacoward wrote: post #510

throw0101a wrote: post #469

Earlier quoted context omitted.

> This supposed need for ZFS to absorb all RAID/LVM/page-cache behavior into itself is a myth; what really happened is good old-fashioned NIH. At the time that ZFS was written (early 2000s) and released to the public (2006), this was not a thing and the idea was somewhat novel / 'controversial'. Jeff Bonwick, ZFS co-creator, lays out their thinking: * https://blogs.oracle.com/bonwick/rampant-layering-violation Rememb…

I debated some of this with Bonwick (and Cantrill who really had no business being involved but he's pernicious that way) at the time. That blog post is, frankly, a bit misleading. The storage "stack" isn't really a stack. It's a DAG. Multiple kinds of devices, multiple filesystems plus raw block users (yes they still exist and sometimes even have reason to), multiple kinds of functionality in between. An LVM API all…

Sorry, I'm pernicious in what way, exactly?

Re: Linus: Don't Use ZFS

#512

Post by awused » Fri, Jan 10, 2020, 9:37 PM UTC

HorstG wrote: post #477

awused wrote: post #449

Earlier quoted context omitted.

That is not over-provisioning, it's just that 'df' doesn't have the concept of pooled storage. With pools it's possible for different file systems to share their "available" space. BTRFS also has its own problems with ouput when using df and getting strange results. If I have a 10GB pool and I create 10 empty file systems, the sizes reported in df will be 100GB. It's not quite a lie either, because each of those 10 f…

This is exactly what overprovisioning is: The sum of possible future allocations is greater than available space.

In my example the sum of possible future allocations for ZFS is still only 10GB total. Each of the ten file systems, considered individually, does truthfully have 10GB available to it before any data is written. The difference is that with over-provisioning (like LVM+XFS), if I write 10GB of data to one file system the others will still report 10GB of free space, but with ZFS or BTRFS they'll report 0GB available, so I can never actually attempt to allocate 100GB of data.

You could build a pool-aware version of DF that reflects this, by grouping file systems in a pool together and reporting that the pool has 10GB available. But frankly there's not enough benefit to doing that because people with storage pools already understand summing up all the available space from df's output is not meaningful. Tools like zpool list and BTRFS's df equivalent already correctly report the total free space in the pool.

Re: Linus: Don't Use ZFS

#513

Post by rleigh » Fri, Jan 10, 2020, 10:00 PM UTC

ptman wrote: post #375

mschuster91 wrote: post #345

Earlier quoted context omitted.

why would you want to embed raid5/6 in the filesystem layer? Linux has battle-tested mdraid for this, I'm not going to trust a new filesystem's own implementation over it. Same for encryption, there are already existing crypto layers both on the block and filesystem (as an overlay) level.

Why does ZFS do RAIDZ in the filesystem layer?

It doesn't.

RAIDZ is part of the VDEV (Virtual Device) layer. Layered on top of this is the ZIO (ZFS I/O layer). Together, these form the SPA (Storage Pool Allocator).

On top of this layer we have the ARC, L2ARC and ZIL. (Adaptive Replacement Caches and ZFS Intent Log).

Then on top of this layer we have the DMU (Data Management Unit), and then on top of that we have the DSL (Dataset and Snapshot Layer). Together, the SPA and DSL layers implement the Meta-Object Set layer, which in turn provides the Object Set layer. These implement the primitives for building a filesystem and the various file types it can store (directories, files, symlinks, devices etc.) along with the ZPL and ZAP layers (ZFS POSIX Layer and ZFS Attribute Processor), which hook into the VFS.

ZFS isn't just a filesystem. It contains as many, if not more, levels of layering than any RAID and volume management setup composed of separate parts like mdraid+LVM or similar, but much better integrated with each other.

It can also store stuff that isn't a filesystem. ZVOLs are fixed size storage presented as block devices. You could potentially write additional storage facilities yourself as extensions, e.g. an object storage layer.

Re: Linus: Don't Use ZFS

#514

Post by the_why_of_y » Fri, Jan 10, 2020, 10:07 PM UTC

lmm wrote: post #220

lmilcin wrote: post #41

Earlier quoted context omitted.

The problem with ZFS is that it isn't part of Linux kernel. Linux project maintains compatibility with userspace software but it does not maintain compatibility with 3rd party modules and for a good reason. Since modules have access to any internal kernel API it is not possible to change anything within kernel without considering 3rd party code, if you want to keep that code working. For this reason the decision was…

It isn't just ZFS. All sorts of drivers get broken because Linux refuses to offer a stable API, saying your code should be in the kernel, but also often refuses to accept drivers into the kernel, even open-source code with no particular quality issues (e.g. quickcam, reiserfsv4). Use FreeBSD where there's a stable ABI and you don't have these problems.

FreeBSD does not really have a stable ABI; every major release breaks the ABI, so it's only stable for 2 years.

https://wiki.freebsd.org/VendorInformation

Re: Linus: Don't Use ZFS

#515

Post by notacoward » Fri, Jan 10, 2020, 10:08 PM UTC

bcantrill wrote: post #511

notacoward wrote: post #510

Earlier quoted context omitted.

I debated some of this with Bonwick (and Cantrill who really had no business being involved but he's pernicious that way) at the time. That blog post is, frankly, a bit misleading. The storage "stack" isn't really a stack. It's a DAG. Multiple kinds of devices, multiple filesystems plus raw block users (yes they still exist and sometimes even have reason to), multiple kinds of functionality in between. An LVM API all…

Sorry, I'm pernicious in what way, exactly?

Heh. I was wondering if you were following (perhaps participating in) this thread. "Pernicious" was perhaps a meaner word than I meant. How about "ubiquitous"?

Re: Linus: Don't Use ZFS

#516

Post by rleigh » Fri, Jan 10, 2020, 10:11 PM UTC

HorstG wrote: post #206

LaGrange wrote: post #154

Earlier quoted context omitted.

> There are no (stable) alternatives. BTRFS certainly not, as it's "under heavy development"¹ (since... forever). Note that they don't mean "it's unstable," just "there are significant improvements between versions." Most importantly: > The filesystem disk format is stable; this means it is not expected to change unless there are very strong reasons to do so. If there is a format change, filesystems which implement t…

Some features such as Raid5 were still firmly in "don't use if you value your data" territory last I looked. So it is important to be informed as to what can be used and what might be more dangerous with btrfs

Btrfs has many more problems than dataloss with RAID5.

It has terrible performance problems under many typical usage scenarios. This is a direct consequence in the choice of core on-disc data structures. There's no workaround without a complete redesign.

It can become unbalanced and cease functioning entirely. Some workloads can trigger this in a matter of hours. Unheard of for any other filesystem.

It suffers from critical dataloss bugs in setups other than RAID5. They have solved a number of these, but when reliability is its key selling point many of us have concerns that there is still a high chance that many still exist, particularly in poorly-exercised codepaths which are run in rare circumstances such as when critical faults occur.

And that's only getting started...

Re: Linus: Don't Use ZFS

#517

Post by rhinoceraptor » Fri, Jan 10, 2020, 10:12 PM UTC

bitL wrote: post #499

rhinoceraptor wrote: post #494

Earlier quoted context omitted.

ZFS needs ECC just as much as any other file system. That is, it has no way of detecting in memory errors. So if you want your data to actually be written correctly, it's a good idea to use ECC. But the myth that you "need" ECC with ZFS is completely wrong. It would be better if you did have ECC, but don't let that stop you from using ZFS. As far as it needing a lot of memory, that is also not true. The ARC will use…

I worked at SUN when ZFS was "invented" and the emphasis on a large amount of proper ECC memory was strong, especially in conjunction with Solaris Zones. I can't recall if it was 1GB of RAM per 1TB of storage or something similar due to how it performed deduplication and stored indices in hot memory. And that was also the reason for insisting on ECC, in order to make sure you won't get your stored indices and shared…

I can see how a (perhaps, less than competitive) hardware company would want you to think that :)

Re: Linus: Don't Use ZFS

#518

Post by rleigh » Fri, Jan 10, 2020, 10:20 PM UTC

m4rtink wrote: post #376

awused wrote: post #286

Earlier quoted context omitted.

>Isn't this a problem for any over provisioned storage pool ? ZFS doesn't over-provision anything by default. The only case I'm aware of where you can over-provision with ZFS is when you explicitly choose to thin provision zvols (virtual block devices with a fixed size). This can't be done with regular file systems which grow as needed, though you can reserve space for them. File systems do handle running out of spac…

Can't you over provision even just by creating too many many snapshots ? Even if you never make the filesystems bigger then the backing pool, the snapshots will allocate some blocks from the pool and over time, boom.

ZFS doesn't work this way. The free blocks in the ZFS pool are available to all datasets (filesystems). The datasets themselves don't take up any space up front until you add data to them. Snapshots don't take up any space initially. They only take up space when the original dataset is modified, and altered blocks are moved onto a "deadlist". Since the modification allocates new blocks, if the pool runs out of space it will simply return ENOSPC at some point. There's no possibility of over-provisioning.

ZFS has quotas and reservations. The former is a maximum allocation for a dataset. The latter is a minimum guaranteed allocation. Neither actually allocate blocks from the pool. These don't relate in any comparable way to how LVM works. They are just numbers to check when allocating blocks.

Re: Linus: Don't Use ZFS

#519

Post by rleigh » Fri, Jan 10, 2020, 10:24 PM UTC

dekhn wrote: post #144

StavrosK wrote: post #136

Earlier quoted context omitted.

Same here (4-drive raidz for many years), though I do have an issue where deleting large files (~1 GB) takes around a minute and nobody seems to know why (I have plenty free space and RAM)...

do you have lots of snapshots? every snapshotting FS I've worked with has really slow deletes, especially when the volume is near capacity.

Snapshots are one thing ZFS is fast at. All the blocks for a given snapshot are placed on a "deadlist". Snapshot deletion is essentially just returning this list of blocks back to the free pool. A terabyte snapshot will take a short while (in the background) to recycle those blocks. But the deletion itself is near instantaneous.

Re: Linus: Don't Use ZFS

#520

Post by rleigh » Fri, Jan 10, 2020, 10:26 PM UTC

the_why_of_y wrote: post #514

lmm wrote: post #220

Earlier quoted context omitted.

It isn't just ZFS. All sorts of drivers get broken because Linux refuses to offer a stable API, saying your code should be in the kernel, but also often refuses to accept drivers into the kernel, even open-source code with no particular quality issues (e.g. quickcam, reiserfsv4). Use FreeBSD where there's a stable ABI and you don't have these problems.

FreeBSD does not really have a stable ABI; every major release breaks the ABI, so it's only stable for 2 years. https://wiki.freebsd.org/VendorInformation

Stable for each major and minor release is still a vast step up on Linux.

Having a stable ABI for two years is vastly easier to support than an ABI which changes every two weeks. This is reflected by the number of binary modules which are packaged for FreeBSD in the ports tree, and provided by third-party vendors. This stability makes it possible to properly support for a reasonable timeframe, and vendors are doing so.