Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A data corruption bug in OpenZFS?

despairlabs.com

Post reply on HN
Print view View original

91–100 of 115 posts

Re: A data corruption bug in OpenZFS?

#91

Post by p_l »

marcinzm wrote: post #77
paldepind2 wrote: post #75

Earlier quoted context omitted.

> Just because Shuttleworth thinks he cant be sued by Oracle for including ZFS in Ubuntu with some hastily reasoned shim doesnt mean Oracle wont nonchalantly send his entire company to the graveyard just for trying. Canonical has been shipping the kernel with ZFS for more than 7 years and so far they have not been sued by Oracle.

Oracle doesn't sue for fun and Canonical isn't exactly a massively successful company financially speaking. However one day it will want something from Canonical and that is the day the lawyers will come out. Possibly once there's IPO money in it's coffers to go after.

Except Oracle has no capability to sue anyone over OpenZFS.

Re: A data corruption bug in OpenZFS?

#92

Post by p_l »

mardifoufs wrote: post #76
nimbius wrote: post #71

Earlier quoted context omitted.

Oracle laid off basically every Solaris developer in 2017. They are by all observation simply not interersted in the product anymore. its probably the most mournful thing ive seen in tech in a very long time. OpenZFS is a mighty filesystem hobbled by an absolutely detestable license (the CDDL.) Its greatest single contribution was in all likelyhood to BSD, although it didnt seem to make the OS more popular as a whole…

How is the CDDL any more detestable than the GPL family of licenses? Not saying that they are detestable in any way, but the CDDL is also a free software license so I don't get how it's worse or bad

Specifically, the issue is with GPL, which disallows licenses that have more requirements than itself - in case of CDDL, it's IIRC patent sharing requirements and language around file-specific applicability.

And of course then there's the part that GPL doesn't apply to linking at all - it applies to derivative code, which OpenZFS is not and thus it does not violate GPL to ship OpenZFS code linked with Linux kernel.

Re: A data corruption bug in OpenZFS?

#93

Post by p_l »

nimbius wrote: post #71
LanzVonL wrote: post #69

It's important to note that the recent showstopper bugs have all been in OpenZFS, with the Oracle nee Sun ZFS being unaffected by either.

Oracle laid off basically every Solaris developer in 2017. They are by all observation simply not interersted in the product anymore. its probably the most mournful thing ive seen in tech in a very long time. OpenZFS is a mighty filesystem hobbled by an absolutely detestable license (the CDDL.) Its greatest single contribution was in all likelyhood to BSD, although it didnt seem to make the OS more popular as a whole…

Oracle isn't copyright holder for OpenZFS. That's one part that OpenSolaris and OpenZFS projects managed to ensure. What Oracle could do was to close OpenSolaris again under proprietary license, something that Brian Cantrill IIRC blamed on the use of copyright assignment, and that open source projects should never use it - with that as a specific example.

OpenZFS devs have openly declared that no, they are not pushing to include OpenZFS into Linux kernel, and that separate arrangement is just fine, especially since it allows different release cadence and keeps code portable.

Mainly there's an issue with certain Linux Kernel big name(s) that like to use GPL-only exports (something that has uncertain legal status) in a rather blunt way, and sometimes the reasoning is iffy.

Re: A data corruption bug in OpenZFS?

#94

Post by avianlyric »

ajross wrote: post #63
avianlyric wrote: post #58

Earlier quoted context omitted.

> Making the filesystem, implemented from first principles to handle the second style of interaction, actually be implemented in terms of the first under the hood, is a source of needless complexity and bugs. And it was here, too. Aren’t all modern file systems implemented as a tree of discontinuous regions? That’s the whole reason block allocators exist, why file fragmentation is a thing (and defragmentation process…

That's true of a storage backend, but not the metaphor presented. Again, the analogy would be a heap: heaps are discontiguous internally too, but you don't demand that users of malloc() understand that there can be a hole in the middle of their memory! Again, the bug here was (seems to have been, it's subtle) a glitch in the tracking of holes in files that didn't ever need to have been there in the first place.

But ZFS doesn't demand that users be aware of holes in files. You can just call `seek()` and `read()` to anywhere, and ZFS will transparently provide zeros to fill the holes. Linux also allows software to become "hole-aware" using `lseek()`, but that's an optimisation that software can opt into, but can equally just ignore.

The glitch in this case was a failure to correctly track dirty pages that have yet to be written to disk, and thus reading the on-disk data, rather than the data in-memory data within the dirty page. I just so happens this issue only appears in the code that's responsible for responding to queries about holes from software that's explicitly asking to know about the holes. ZFS itself never had any issues keeping track of the holes, the bookkeeping always converged on the correct state, it's just that during that convergence it was momentarily possible to be given old metadata about holes (i.e. what's currently on disk), rather than the current metadata about holes (i.e. what's currently only in-memory, and about to be written to disk).

Re: A data corruption bug in OpenZFS?

#95

Post by avianlyric »

p_l wrote: post #80
avianlyric wrote: post #58

Earlier quoted context omitted.

> Making the filesystem, implemented from first principles to handle the second style of interaction, actually be implemented in terms of the first under the hood, is a source of needless complexity and bugs. And it was here, too. Aren’t all modern file systems implemented as a tree of discontinuous regions? That’s the whole reason block allocators exist, why file fragmentation is a thing (and defragmentation process…

The issue is that pretty much all other filesystems at least on Linux, are effectively implemented as swap filesystem drivers with some hierarchical structure on top, because that's the interface pushed by Linux at kernel level . In userland, we tend to think of streams of bytes, as provided by original Unix and as all the docs teach us to treat them - that read(), write() are the primitives and they do byte-aligned…

What any of that got to do with the bug described in the article? Presumably every filesystem is responsible for tracking the content of sparse files, and where holes are. That's not something the Linux kernel is going to give you for free, the FS needs tell the kernel which pages should be mapped to block address on disk and which pages should be simulated as continuous blocks of zeros with no on-disk representation.

Re: A data corruption bug in OpenZFS?

#96

Post by avianlyric »

loeg wrote: post #62
wolf550e wrote: post #50

Earlier quoted context omitted.

Imagine a torrent client (or http client downloading a file in parallel using HTTP range requests). It creates an empty file and then it has downloaded a 1MB of data to write at offset 100GB and wants to write it to disk. It does not want to pay the price of waiting for 100GB of zeroes to be written. The other blocks will all be downloaded and written eventually, all out of order. If the filesystem had an atomic oper…

Fallocate is a much better interface for this than sparse files. The torrent client does not care how the underlying filesystem provides the ability to randomly write a large file. And fallocate is a much clearer signal to the filesystem than a sparse file.

fallocate is just an interface to create sparse files. The result of using `fallocate` is a sparse file.

Re: A data corruption bug in OpenZFS?

#97

Post by avianlyric »

nimbius wrote: post #71
LanzVonL wrote: post #69

It's important to note that the recent showstopper bugs have all been in OpenZFS, with the Oracle nee Sun ZFS being unaffected by either.

Oracle laid off basically every Solaris developer in 2017. They are by all observation simply not interersted in the product anymore. its probably the most mournful thing ive seen in tech in a very long time. OpenZFS is a mighty filesystem hobbled by an absolutely detestable license (the CDDL.) Its greatest single contribution was in all likelyhood to BSD, although it didnt seem to make the OS more popular as a whole…

> You'll see a stable BTRFS and a continued effort to polish XFS/LVM/MDRAID before openZFS ever makes a dent.

Right now I would put my money on bcachefs[1] rather than BTRFS. bcachefs is currently in the process of being merged into the kernel and will be in the next kernel release. Doesn't currently quite offer everything ZFS does, but it's very close and already appears more reliable than BTRFS, and once stuff like Erasure Coding is stable, it'll be more flexible than ZFS.

[1] https://bcachefs.org

Re: A data corruption bug in OpenZFS?

#98

Post by bb88 »

rustcleaner wrote: post #73
nimbius wrote: post #71

Earlier quoted context omitted.

Oracle laid off basically every Solaris developer in 2017. They are by all observation simply not interersted in the product anymore. its probably the most mournful thing ive seen in tech in a very long time. OpenZFS is a mighty filesystem hobbled by an absolutely detestable license (the CDDL.) Its greatest single contribution was in all likelyhood to BSD, although it didnt seem to make the OS more popular as a whole…

This is bull**! It's time for a new license to throw off the old: The Uniform Pirating License! It's a license which you stick on anything which you need a license, and it conveys all rights and zero obligations to you. Possession of the code is sufficient to run, change, and propagate code. Legal system be damned; we have cryptography and Tor, The State's law here is irrelevant (also when did you give The State lice…

There are some very creative solutions for getting around license restrictions.

The LAME mp3 encoder originally was a series of patches that could be applied to the Fraunhaufer ISO dist10 release.

Re: A data corruption bug in OpenZFS?

#99

Post by loeg »

avianlyric wrote: post #96
loeg wrote: post #62

Earlier quoted context omitted.

Fallocate is a much better interface for this than sparse files. The torrent client does not care how the underlying filesystem provides the ability to randomly write a large file. And fallocate is a much clearer signal to the filesystem than a sparse file.

fallocate is just an interface to create sparse files. The result of using `fallocate` is a sparse file .

You should read my comment in the context of the one it is replying to. That comment suggested a torrent client using seeks + writes to randomly insert chunks as they were downloaded. I have summarized this approach in my comment as "sparse files," expecting charitable readers to be familiar with the context. This method of creating sparse files does not tell the filesystem anything about the intent of the application and usually creates a bunch of fragmentation under torrent-like workloads.

Re: A data corruption bug in OpenZFS?

#100

Post by epcoa »

loeg wrote: post #99
avianlyric wrote: post #96

Earlier quoted context omitted.

fallocate is just an interface to create sparse files. The result of using `fallocate` is a sparse file .

You should read my comment in the context of the one it is replying to. That comment suggested a torrent client using seeks + writes to randomly insert chunks as they were downloaded. I have summarized this approach in my comment as "sparse files," expecting charitable readers to be familiar with the context. This method of creating sparse files does not tell the filesystem anything about the intent of the applicatio…

It’s not about being charitable. For those unfamiliar with the terminology this is just confusing, and for those that are familiar this discussion is all fundamental and well known anyway.

Unfortunately for COW filesystems including zfs and btrfs fallocate doesn’t do anything useful for preallocation. You’re still going to get fragmentation. The two methods outlined are essentially equivalent.

Post reply on HN